OpenGL/GLUT - Project ModelView Coordinate to Texture Matrix

Question

Is there a way using OpenGL or GLUT to project a point from the model-view matrix into an associated texture matrix? If not, is there a commonly used library that achieves this? I want to modify the texture of an object according to a ray cast in 3D space.

The simplest case would be:

1. A ray is cast which intersects a quad, mapped with a single texture.
2. The point of intersection is converted to a value in texture space clamped between [0.0,1.0] in the x and y axis.
3. A 3x3 patch of pixels centered around the rounded value of the resulting texture point is set to an alpha value of 0.( or another RGBA value which is convenient, for the desired effect).

To illustrate here is a more complex version of the question using a sphere, the pink box shows the replaced pixels.

I just specify texture points for mapping in OpenGL, I don't actually know how the pixels are projected onto the sphere. Basically I need to to the inverse of that projection, but I don't quite know how to do that math, especially on more complex shapes like a sphere or an arbitrary convex hull. I assume that you can somehow find a planar polygon that makes up the shape, which the ray is intersecting, and from there the inverse projection of a quad or triangle would be trivial.

Some equations, articles and/or example code would be nice.

Answer 1

So I found a solution that is a little complicated, but does the trick.

For complex geometry you must determine which quad or triangle was intersected, and use this as the plane. The quad must be planar(obviously).

Draw a plane in the identity matrix with dimensions 1x1x0, map the texture on points identical to the model geometry.

1. Transform the plane, and store the inverse of each transform matrix in a stack
2. Find the point at which the the plane is intersected
3. Transform this point using the inverse matrix stack until it returns to identity matrix(it should have no depth(
4. Convert this point from 1x1 space into pixel space by multiplying the point by the number of pixels and rounding. Or start your 2D combining logic here.

Answer 2

There are a few ways you could accomplish what you're trying to do:

1. Project a world coordinate point into normalized device coordinates (NDCs) by doing the model-view and projection transformation matrix multiplications by yourself (or if you're using old-style OpenGL, call gluProject), and perform the perspective division step. If you use a depth coordinate of zero, this would correspond to intersecting your ray at the imaging plane. The only other correction you'd need to do map from NDCs (which are in the range [-1,1] in x and y) into texture space by dividing the resulting coordinate by two, and then shifting by .5.

2. Skip the ray tracing all together, and bind your texture as a framebuffer attachment to a framebuffer object, and then render a big point (or sprite) that modifies the colors in the neighborhood of the intersection as you want. You could use the same model-view and projection matrices, and will (probably) only need to update the viewport to match the texture resolution.