In what coordinate system do normal vertices end up after transformation by a normal matrix?

Question

In computer graphics, normal vectors are used to determine the direction in which the surface of some geometry is 'facing' because they're perpendicular to said surface.

When a vertex's position is transformed by a model-view matrix, the vertex is now said to be in "view-space", because it's now in a coordinate system that's relative to the viewer/camera.

However, when a vertex normal is transformed, a normal matrix¹ is used instead.

While I understand why this is done, I'm not sure if the transformed normal vertex is also said to be in "view-space". Intuitively, it seems like this might be the case.

Is it actually correct to say that a vertex normal transformed by a normal matrix is in "view space"? If not, is the normal vertex transformed into the same coordinate space, or is there some other, more appropriate, term?

---

¹ The transpose of the inverse of the model-view matrix.

Answer 1

Yes. Let me explain. (sorry about writing an essay!)

While both vertices and normals are usually represented as floating point 3D vectors, their nature makes them different.

A coordinate space is nothing more than a comparison of which origin geometry is measured relative to.

For example, I define two cubes sized 1x1x1. The origin from which all their 8 vertices are defined is at their very centre. Their vertex coordinates are therefore all possible combinations of (+-0.5, +-0.5, +-0.5).

These cubes may seem to have similar coordinates, but we have thus far only considered them as objects on their own. To place them in a scene, we need to define a transformation which will define their location, orientation and size in that scene.

If I apply the identity transformation on both cubes, I can now say they are in the same coordinate space, because the origins by which the coordinates of all vertices of both cubes are defined is now the same.

It doesn't matter if I change either or both transformations to whichever transformation I please. As long as I can say that after the transformation is applied on the coordinates of each respective cube, the coordinates of all vertices of each cube is measured relative to the same origin, they are in the same coordinate space.

---

So how do normals fit in this picture?

I'll make a distinction between a surface normal and a normal vector here. A surface normal is a mathematical concept representing a normalised vector orthogonal to a particular point on the surface of a model. A normal vector is the specific value of this surface normal at a point that is known. The normal vector is what you usally store in memory along with your vertex coordinates.

Some transformations can change the surface normal of a model. Others do not. If I translate any of the cubes I defined previously, the directions of each surface of the cube does not change either. Therefore the surface normals remain exactly the same whuchever point you choose on the surface. The same is true when scaling the cube.

However, if I rotate a cube, the orientation of some or all surfaces on the cube changes. This means in turn the surface normals change. The normal vectors stored as part of the vertex specifications in memory are thus outdated and must be rotated as well to make them consistent again with the surface normals.

We can thus say that normal vectors are dependent on surface normals, which in turn are dependent on any transformation done on the vertices (or any point on the surface of a model). Therefore, applying a transformation on a vertex, taking it from one coordinate space to another, causes the normal vector to follow suit.

Even when the surface normal is technically not always affected by a transformation applied on a model (see translation), a transformation from one coordinate space to another can involve only an identity transformation, as I've shown previously.

So to answer your question: Yes, applying a series of potentially different transformations on a set of potentially different coordinates transforming them into the same coordinate space will potentially change their normals too which will regardless be considered to be in the same coordinate space as the vertices / point on the surface of a model of which they represent the surface normal.