Android OpenCV Image Rectification

Question

i'm tryin to do image rectification without calibration , just using fundamental Matrix and using OpenCV Here things that i've done till now :

1. Find Matches between two images using SURF and FLANN Descriptor Matcher

   List good_match = getGoodMatch(img1,img2); --> it's done , and the result is good

2. Get Matching Point for img1 and img2

3. Get Fundamental Matrix using 8 Matches Point (Calib3D.FM_8PPoint) Mat F = getFundamentalMatrix(matchesKeyPoint1,matchesKeyPoint2,good_match)
4. Get Homography 3x3 Matrix using StereorectufyUncalibrated OpenCV

i've got the Homography Matrix H1 and H2 from StereoRectifyUncalibrated, so how to use that homography matrix to rectify my real images?? and is there any way to know that Fundamental Matrix and Homography Matrix have good value?? So far the SURF Image Matching did his job well. And is there any suggestion to improve the result??

thanks...

Answer 1

Rectify the images

Given a set of 2D point correspondences X <-> X', the transformation of a homography matrix is H given by X' = H X. In here X and X' are homogeneous vectors, which means that the 3D vectors X' and H X do not have to be equal, but they can differ in magnitude by a non zero scale factor.

So basicly what we would like to do is multiply every pixel in an image with the homography matrix. Also, we would like to apply some kind of interpolation between the transformed pixels such that we get a smooth image without 'empty' pixels. Luckily, such a function exists in OpenCV: cvWarpPerspective.

This function (in OpenCV 2.0) requires 4 parameters:

• const CvArr* src - The source image, i.e. the image you would like to rectify.
• CvArr* dst - The destination image, i.e. the result after rectification. This image should be the same type of image as the source image. And the size should be such that it fits the rectified image (see next paragraph).
• const CvMat* mapMatrix - The 3 X 3 homography matrix.
• int flags - A combination of an interpolation method (CV_INTER_LINEAR for instance) and the flags CV_WARP_FILL_OUTLIERS and CV_WARP_INVERSE_MAP.

To determine the size of the destination image, you can apply the transformation of the homography matrix on the corners of the source image and uses the transformed points to determine the size. The transformation for the top left corner will be as follows:

CvMat* point = cvCreateMat(3, 1, CV_64FC1);
CvMat* pointTransform = cvCreateMat(3, 1, CV_64FC1);

cvmSet(point, 0, 0, 0); // The x coordinate of the top left corner is 0.
cvmSet(point, 1, 0, 0); // The y coordinate of the top left corner is 0.
cvmSet(point, 2, 0, 1.0);

// Perform the homography transformation
cvMatMul(homography, point, pointTransform);

// Get the transformed corner's x and y coordinates.
double x = cvmGet(pointTransform, 0, 0) / cvmGet(pointTransform, 2, 0); // Divide by the scale factor s.
double y = cvmGet(pointTransform, 1, 0) / cvmGet(pointTransform, 2, 0); // Divide by the scale factor s.

// Release memory
cvReleaseMat(&point);
cvReleaseMat(&pointTransform);

Next thing you should notice is that a pixel (0, 0) could be transformed to a pixel (-5, -10) for instance. If you would apply the homography matrix and then try to display the image it would not show properly. The thing you should do to avoid this is compute a 3 x 3 translation matrix, based on the position of the new corners of the rectified image. The corners will give you the information how much pixels you have to shift your rectified image up or down and left or right.

Then you can use this translation matrix in combination with your found homography matrix to compute the final transformation matrix as follows:

// transformMat:
// [ 1 0 x ]
// [ 0 1 y ]
// [ 0 0 1 ]
// homography: Your homography matrix H1 or H2
// finalMatrix: the resulting matrix you will use in cvWarpPerspective.
// Compute the final transformation matrix based on homography matrix H1
// which can be used to rectify your first image.
cvMatMul(transformMat, H1, finalMatrix);

Now you can use cvWarpPerspective to transform your image:

cvWarpPerspective(image, rectifiedImage, finalMatrix, 
CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS);

Note you have use homography matrix H1 for your first image and homography matrix H2 for your second image.

Do the fundamental matrix and homography matrices have good values?

Furthermore, to answer your question to see if the Fundamental matrix F and the Homography matrices H1 and H2 have good values. The fundamental matrix F should satisfy the condition that for any pair of corresponding points X <-> X':

X' F X = 0.

Similarly, for matrix H1 the following should be satisfied:

X' cross H1 X = 0, where 'cross' is the cross product.

Improving your results

To improve your results further you could:

• Use a different descriptor to find matches between two images. For example see SIFT and DAISY.
• Find more than 8 point correspondences between two images and filter outliers using a RANSAC or LMS algorithm (see the options CV_FM_RANSAC and CV_FM_LMEDS in cvFindFundamentalMatrix).