CODEWITHSUNDEEP
camerarotational-matricespose-estimationopencv-solvepnp
dudu
dudu

Reputation: 849

pose estimation: determine whether rotation and transmation matrix are right

Recently I'm struggling with a pose estimation problem with a single camera. I have some 3D points and the corresponding 2D points on the image. Then I use solvePnP to get the rotation and translation vectors. The problem is, how can I determine whether the vectors are right results?

Now I use an indirect way to do this:
I use the rotation matrix, the translation vector and the world 3D coordinates of a certain point to obtain the coordinates of that point in Camera system. Then all I have to do is to determine whether the coordinates are reasonable. I think I know the directions of x, y and z axes of Camera system.

The figure below is in world space, while the axes depicted are in Camera system.

========How Camera and the point be placed in the world space=============
   |
   |              
Camera--------------------------> Z axis
   |                |} Xw?
   |                P(Xw, Yw, Zw)
   |              
   v x-axis

Picture here

My rvec and tvec results seems right and wrong. For a specified point, the z value seems reasonable, I mean, if this point is about one meter away from the camera in the z direction, then the z value is about 1. But for x and y, according to the location of the point I think x and y should be positive but they are negative. What's more, the pattern detected in the original image is like this:

pattern image using feature detection from the camera captured image

But using the points coordinates calculated in Camera system and the camera intrinsic parameters, I get an image like this:

recovered image

The target keeps its pattern. But it moved from bottom right to top left. I cannot understand why.

Upvotes: 1

Views: 1921

Answers (2)

dudu
dudu

Reputation: 849

Now I know the answers.

  • Yes, the camera center is the origin of the camera coordinate system.

  • Consider that the coordinates in the camera system are calculated as (xc,yc,zc). Then xc should be the distance between the camera and the point in real world in the x direction.

Next, how to determine whether the output matrices are right?
1. as @eidelen points out, backprojection error is one indicative measure.
2. Calculate the coordinates of the points according to their coordinates in the world coordinate system and the matrices.

So why did I get a wrong result(the pattern remained but moved to a different region of the image)?
Parameter cameraMatrix in solvePnP() is a matrix supplying the parameters of the camera's external parameters. In camera matrix, you should use width/2 and height/2 for cx and cy. While I use width and height of the image size. I think that caused the error. After I corrected that and re-calibrated the camera, everything seems fine.

Upvotes: 2

Adrian Schneider
Adrian Schneider

Reputation: 1507

Yes, the camera center is the origin of the camera coordinate system, which seems to be right following to this post.

In case of camera pose estimation, value seems reasonable can be named as backprojection error. That's a measure of how well your resulting rotation and translation map the 3D points to the 2D pixels. Unfortunately, solvePnP does not return a residual error measure. Therefore one has to compute it:

cv::solvePnP(worldPoints, pixelPoints, camIntrinsics, camDistortion, rVec, tVec);

// Use computed solution to project 3D pattern to image
cv::Mat projectedPattern;
cv::projectPoints(worldPoints, rVec, tVec, camIntrinsics, camDistortion, projectedPattern);

// Compute error of each 2D-3D correspondence.
std::vector<float> errors;
for( int i=0; i < corners.size(); ++i)
{
    float dx = pixelPoints.at(i).x - projectedPattern.at<float>(i, 0);
    float dy = pixelPoints.at(i).y - projectedPattern.at<float>(i, 1);
    // Euclidean distance between projected and real measured pixel
    float err = sqrt(dx*dx + dy*dy); 

    errors.push_back(err);
}

// Here, compute max or average of your "errors"

An average backprojection error of a calibrated camera might be in the range of 0 - 2 pixel. According to your two pictures, this would be way more. To me, it looks like a scaling problem. If I am right, you compute the projection yourself. Maybe you can try once cv::projectPoints() and compare.

When it comes to transformations, I learned not to follow my imagination :) The first thing I Do with the returned rVec and tVec is usually creating a 4x4 rigid transformation matrix out of it (I posted once code here). This makes things even less intuitive, but instead it is compact and handy.

Upvotes: 2

Related Questions