Correctly transforming a surface normal

Question

According to the OpenGL Red Book,Appendix F, a regular 3D transformation matrix M can be used to calculate the action on a normal vector as:

normalTransformed = transpose(inverse(M)) * normal

However, while the orthogonal plane associated with the transformed normal is indeed parallel to the transformed surface, it can happen that the transformed normal vector itself is pointing in the opposite direction of what I would expect, i.e., 'into' the surface rather than 'out of' the surface.

If I want the normalTransformed to point in the correct direction (i.e., the same direction as it points to when the surface to which it is attached is not transformed), how should I do that, mathematically?

EXAMPLE

Suppose my surface normal is (0,0,1), and my transform is a translation by 10 in the Z direction. The transformation matrix M is then:

1 0 0 0
0 1 0 0
0 0 1 10
0 0 0 1

The transpose(inverse(M)) is then:

1 0 0 0
0 1 0 0
0 0 1 0
0 0 -10 1

Applied to the surface normal (0,0,1), i.e., (0,0,1,1) in homogeneous coordinates, this gives:

normalTransformed = (0, 0, 1, -9)

Back from homogeneous coordinates:

(0, 0, -1/9)

Normalizing to length 1:

(0, 0, -1)

Which points to the opposite direction compared to the original normal vector (0, 0, 1).

Answer 1

You should only get a reversal in the relative direction of the normal if the affine transformation you are applying reverses the "handedness" of your coordinate system. This would happen for example if you scaled by [1, 1, -1].

According to the book Physically Based Rendering you can check for this case by calculating the determinant of the upper left 3x3 (normal?) matrix. If the determinant is negative then the matrix will change the handedness, and you should invert the normal.

[I was just reading about this yesterday, and am quoting this from memory].

Answer 2

The problem is that you are dividing by your w coordinate, as if your normal were a point. (When w<0, this division is what is reversing your normal.) Instead, you need to completely ignore the w-coordinate: drop it entirely, instead of dividing by it.

Your normal is not a point, it is technically a covector (which is why it is transformed differently from points and vectors). It doesn't actually have a w-coordinate -- the only reason to add one is for convenience in using existing 4x4 matrix routines.

If you do add an arbitrary w-coordinate, you have the homogeneous coordinates for a plane with the given normal. Like the normal, such a plane is is transformed by the inverse-transpose of the matrix used to transform points (and, note that dividing a plane by its w-coordinate doesn't make sense -- a plane is not a point, either!).

If the normal was derived from a triangle, the plane of the triangle should have that normal -- however, a normal explicitly lacks the w-coordinate that determines exactly which plane that is. Adding an arbitrary w to a normal (whether it be 0, 1, or something else) means choosing an arbitrary plane with that normal, so transforming it will yield an arbitrary plane with the transformed normal; this is why you need to ignore the w after transforming with a 4x4 matrix.

Answer 3

What Nicol Bolas has written in his answer is absolutely correct, indeed I won't repeat those concepts.

But I noted in the question a couple of point that can be interesting.

First, the normal matrix is normally defined as the transpose of the inverse of the upper-left 3x3 matrix of the modelview matrix. This is because normals are not defined using homogeneous position, indeed a 4x4 matrix is not necessary; if you want use the entire modelview matrix, follow Nicol Bolas direction, but the math remains the same. (since w is zero).

Second, you have stated

I want the normalTransformed to point in the correct direction (i.e., the same direction as it points to when the surface to which it is attached is not transformed), how should I do that, mathematically?

The normal matrix is used to transform the normal coherently with model transform (infact the normal matrix is derived from the modelview matrix). As I can understand from your quote is that you want the normal not transformed... Indeed, why you transform it? You can use 'normal' directly.

Answer 4

Applied to the surface normal (0,0,1), i.e., (0,0,1,1) in homogeneous coordinates

OK, stop right there.

If you're going to look at surface normals as homogeneous coordinates, you use a zero as the W component, not a 1. Now, you would probably realize fairly quickly that you can't divide by zero, but that's also why you don't do homogeneous math on normals.

A normal is not a position; it is a direction. Directions do not have a position, so translating them is meaningless. A homogeneous position with a W=0 represents a "position" infinitely far away (which is why you can't divide them). A position infinitely far away is infinitely far away from every finite point.

And therefore, a position at infinity is a direction: it doesn't change direction no matter what (finite) position you look at it from.

Now, if you have a 4x4 matrix and need to transform a normal by it, you only use W=0 because it makes the math work out. It gets rid of the translation component of the matrix. The post-transform W component should be completely ignored.

Therefore, after transforming, you get this:

normalTransformed = (0, 0, 1, -9)

Which after ignoring the W component, becomes:

normalTransformed = (0, 0, 1)

---

It is far more likely that your normal is not actually pointed in the right direction to begin with. Of course, in the absence of code and data, little more can be said, but the math works assuming the inputs are legitimate.

Also, don't do the inverse/transpose in the shader. Do it on the CPU and pass the resulting matrix to the shader.