A Stereo Vision-based Mixed Reality System with Natural Feature Point Tracking

This paper proposes a method to extend a registration range of a vision-based mixed reality system. We propose to use natural feature points contained in images captured by a pair of stereo cameras in conjunction with pre-defined fixed fiducial markers. The system also incorporates an inertial sensor to achieve a robust registration method which can handle user’s fast head rotation and movement. The system first uses pre-defined fiducial markers to estimate a projection matrix between real and virtual coordinate systems. At the same time, the system picks up and tracks a set of natural feature points from the initial image. As a user moves around in MR environment, the initial markers fall out from the camera frame and natural features are then used to recover a projection matrix. Experiments evaluating the feasibility of the method are carried out and show the potential benefits of the method.