Image matching using intrinsic and extrinsic camera parameters

Question

Currently I have a dataset of images (sequence of frames) and I have the intrinsic camera calibration matrix. Also, for each image I have the extrinsic parameters (rotation and traslation).

I would like to know if it is possible use that parameters to find the correct pixel correspondences between each pair of images.

I found the relationship traslation (t) and rotation (R) with each correspondence point between two different perspectives.

I guess that using the image above, it is only necessary to fix a "x" point (in homogeneous coordinates) and solve the equation system for "x'", but I do not know what operation is using (notations). If someone know how to do it using matlab, I hope some help.

Also, if there is another way to discover the matching using the same information I hope the help of someone.

Thanks

Answer 1

No, this information is not enough to find point correspondences between the frames. I will first explain what I think that you can do with the given information, and then we'll see why it's impossible to get pixel to pixel matches from the Essential alone.

• What you can do. For a point m, you can find the line on the other image where m' lies, by using the Fundamental matrix. Let's assume that the X and X' you give in your question are (respectively) projected to m and m', i.e.

  //K denotes the intrinsics matrix
  m=KX
  m'=KX'
  

  Starting with your equation, we have:

   X^{T}EX'=0  ==>  m^T K^{-T} E K^{-1} m' 
  

  The matrix K^{-T} E K^{-1} , that we will note F, is known as the Fundamental matrix, and now you have a constraint between 2d points in the image plane:

   m^TFm'=0
  

  Note that m and m' are 3d vectors expressed in homogeneous coordinates. The interesting thing to notice here, is that Fm' is the line on which m lies on the first image (since the constraint given above is nothing but the dot product between m and Fm'). Similarly, m^TF is the line on the other image in which m' is expected to lie. So, what you can do to find a match for m, is to search in a neighborhood of Fm'.

• Why you can't get pixel to pixel matching. Let's look at what the constraint xEx'=0 means from an intuitive point of view. Basically, what it says is that we expect x, x' and the baseline T to be coplanar. Assume that you fix x, and that you look for points that satisfy xEx'=0. Then while the x' in you figure satisfies this constraint, every point n (reprojected from y) such as the one is the figure below will also be a good candidate:

  

  which indicates that the correct match depends on your estimation of the depth of x, which you don't have.