Do Face culling in opengl happens in window space?

Question

The output values of Vertex post processing is in Window space, then we have primitive assembling and Face culling stages.If so Face culling happens in window space?

Answer 1

Let me quote the OpenGL 4.6 core profile specification, section 1.2.3 "What is the OpenGL graphics system? | Our View" (emphasis mine):

We view OpenGL as a pipeline having some programmable stages and some statedriven fixed-function stages that are invoked by a set of specific drawing operations. This model should engender a specification that satisfies the needs of both programmers and implementors. It does not, however, necessarily provide a model for implementation. An implementation must produce results conforming to those produced by the specified methods, but there may be ways to carry out a particular computation that are more efficient than the one specified.

This means that implementations are conforming if they achieve the same results as an implementation which exactly follows the spec word for word would get.

So face culling has to behave the same way as if you would do it in window space. It obviously has the vertex shader stage, since before that, the final coordinates of the vertices are unknown.(Now there is an interesting side note here. It was iirc nvidia which has a patent on splitting the vertex shader in two parts: on which only calculates gl_Position and can optimize out all calculations which do not contribute yo that, and one for the rest of the outputs. This would allow them to do clipping and culling with only having parts of the VS executed. Not sure if they actually employ that strategy in their drivers, but the patent is out there somewhere. However, it is a nice example for what weird tricks real-world implementations might do.)

So does face culling happen in window space? Only your OpenGL implementor knows for sure. But putting the question the other way: Can face culling be implemented to happen in clip space?

First, let's look if it can happen in normalized device space:

Obviously, for face culling, only the winding order of the 2D projection of the triangle matters, so z does not matter. And this orientation doesn't change when you go from clip space to window space, or does it? A few observations here:

• The NDC to window space conversion is just a scale and translation which works on each component individually, we do not get cross-influences of the form x_win = A * x_ndc + B *y_ndc + c * z_ndc + D, but B and C are guaranteed to be 0 here.
• So the only way to flip the triangle orientation when going from clip space to window space would be if the scale factor is negative, and quoting the GL 4.6 spec again on the glViewport*() function family: "An INVALID_VALUE error is generated if either w or h is negative"

This means the face culling could be done in NDC, without having to take the actual viewport parameters into account.

Can the face culling happen in clip space?

• Any point w<0 can never fulfill the clipping condition -w <= x,y,z <= w, so the division by w step to get from clip space to NDC will only happen with w>0 and so never flip the triangle around. But this requires that at least primitive clipping has already happened (consider a case where one vertex lies behind the camera, and two in front of it).
• However, the w coordinate is still tricky even after clipping. Consider the case where you stand in a room with a sloped ceiling and look straight forward onto a side wall, but see part of the ceiling. A triangle from the ceiling could lead to the following clip space coordinates (-1,1,z_1,1), (1,1,z_2,1), (0,1.2,z_3,10) (with some z_is fulfilling the clipping condition, actual values don't matter here). If you just take the clip space x and y values (as you would do in window space or NDC), you would see a 2D projection with the third vertex above the line formed by the first two. But after the division by w, it will be below, and the triangle order is flipped (since w can vary per vertex).

So does this mean that the face culling cannot be done before the perspective divide by w? No, not necessarily. Maybe there is some clever way to carry out the culling in clip space even before the divide, but either way, the clip space w coordinate needs to be taken into account somehow.

So I can't really answer your question, but the general principle is that implementors try to do any sort of culling / discarding steps as early in the pipeline as possible (within all the gazillion engineering constraints they will have, of course).