CODEWITHSUNDEEP
openglcoordinatesraycastingopenscenegraphmatrix-inverse
vicrion
vicrion

Reputation: 1733

OpenSceneGraph: Trying to extract far and near points from mouse screen coordinates for ray casting

I'm trying to do ray casting using OpenSceneGraph. The context of the problem is that I need to draw a line segment on a virtual plane by mouse. I cannot use LineIntersector because the plane does not contain any drawables (I use osg::Plane to represent the virtual plane). I am stuck at the part of how to extract far and near points of the clicked mouse coordinates (my results are not correct).

This is the part of the code that I use. First, calculate the inverse transform:

osg::Matrix VPW = camera->getViewMatrix() *
            camera->getProjectionMatrix() * 
             camera->getViewport()->computeWindowMatrix();

osg::Matrix invVPW;
invVPW.invert(VPW);

Then I calculate the near and far points based on mouse coordinates; dx and dy are in the range [-1 1]: 

double dx = ea.getXnormalized();
double dy = ea.getYnormalized();

ea is of type osgGA::GUIEventAdapter &ea.

Then I multiply the obtained inverse matrix with the mouse coordinates, setting z coordinate to the range representing "far" and "near":

osg::Vec3f nearPoint = osg::Vec3f(dx, dy, 0.f) * invVPW;
osg::Vec3f farPoint = osg::Vec3f(dx, dy, 1.f) * invVPW;

The result point values do not seem to be correct. While the camera is obviously pointing strictly along the y direction [0 1 0], if I take a direction from near to far, the result vector does not make much sense to me, for example,[-0.310973 0.918127 -0.24564].

I'm new in OpenSceneGraph (or OpenGL) and would appreciate any hint of what I am doing wrong. I am not totally sure if it is correct to use normalized screen coords here, however, pixel coordinates did not produce the right result neither.

Upvotes: 0

Views: 1087

Answers (1)

vicrion
vicrion

Reputation: 1733

After a bit of debugging, I was able to figure out where the problem was. First of all, I didn't interpret the result far and near points correctly. The direction of the ray vector also does not represent what I expect to see. It all became more clear after I plugged in my far and near points into an actual ray casting and intersection with my plane.

Second of all, I need to use integer pixel coordinates, not normalized ones. It's because the inverse takes into account the window matrix. If it's ever going to help to anyone, here is the final code I use to extract far and near points using OpenSceneGraph:

osg::Matrix VPW = camera->getViewMatrix()
        * camera->getProjectionMatrix()
        * camera->getViewport()->computeWindowMatrix();

osg::Matrix invVPW;
bool success = invVPW.invert(VPW);

osg::Vec3f nearPoint = osg::Vec3f(ea.getX(), ea.getY(), 0.f) * invVPW;
osg::Vec3f farPoint = osg::Vec3f(ea.getX(), ea.getY(), 1.f) * invVPW;

Upvotes: 0

Related Questions