Is gl_FragDepth equal gl_FragCoord.z when msaa enable?

Question

I know gl_FragDepth will take the value of gl_FragCoord.z from opengl wiki. https://www.khronos.org/opengl/wiki/Fragment_Shader/Defined_Outputs

But I have a problem. If I enable MSAA and write gl_FragDepth = gl_FragCoord.z in fragment shader, the display will not work fine. You can see a black line on the white triangle as below:

If I don't write gl_FragDepth in fragment shader, it will works fine.

If I disable MSAA, it also works fine no matter if I write gl_FragDepth.

The correct display image has no black line:

The render scene is easy, I just draw 2 white triangles and they are intersected on an edge.

I add a simple light in vertex shader. The codes show as below:

const char *vertexShaderSource[] = {
    "#version 120\n",
    "varying vec4 lightColor;\n",
    "void main()\n",
    "{\n",
    "   vec3 n = normalize(gl_NormalMatrix * gl_Normal);\n",
    "   vec3 l = normalize(vec3(0.0, 1.0, 1.0));\n",
    "   float NdotL = clamp(dot(n, l), 0.001, 1.0);\n",
    "   lightColor = vec4(1.0)*(NdotL + 0.2);\n",
    "   gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;\n",
    "}\n"
};

const char *fragmentShaderSource[] = {
    "#version 120\n",
    "varying vec4 lightColor;\n",
    "void main(void)\n",
    "{\n",
    "   gl_FragColor = vec4(lightColor.rgb, 1.0);\n",
    "   gl_FragDepth = gl_FragCoord.z;\n"
    "}\n"
};

The positions of 6 vertices of 2 triangles are (-5,-5,0),(5,-5,0),(-5,5,0),(-5,0,0),(5,0,0),(-5,0,-10). The normals are perpendicular to triangles.

I wanna know why the display images are different if I write gl_FragDepth in fragment shader?

Answer 1

Your two triangles intersect. Specifically, the grey triangle has an edge which can generate depth values equal to the depth values of the white triangle. As such, it is entirely possible for a particular sample from the grey triangle at that intersection to generate a depth value that is equal to the depth value of the white triangle.

So you were never guaranteed to not see a line there; you just happened not to many cases.

However, that all assumes that:

1. The grey triangle is being rendered after the white one.

2. Your depth test will pass on equal values.

The result you are getting here may happen even outside of these two conditions. The reason for that is complex.

See, the whole point of multisampling is that the number depth values generated by the rasterizer and the number of fragment shader executions are not the same. So a single FS invocation is mapped to multiple depth values.

However, a single FS invocation can still write to gl_FragDepth. If it does this, then all samples that map to that FS invocation will receive the same depth. This depth overrides the multisample-generated depth values.

Also, interpolation at the edges of a primitive is weird under multisampling. Each sample that is within the bounds of the triangle at that pixel will result in a sample value being written (unless something else culls it out). But the center point of the pixel need not be one of these sample locations. So a triangle that doesn't pass through the center of a pixel can still contribute some samples to the pixel, so long as the triangle passes through at least one sample in that pixel.

The fragment shader gets interpolated values based on some location inside the pixel. With multisampling, this location may not be inside of the triangle. For example, if the location the implementation selects for the FS's interpolation within the pixel is in the center of the triangle, and the triangle doesn't pass through the center of that pixel, you will still get an FS invocation so long as it passes through some sample.

But this means that the interpolated values can represent locations outside of the area of the triangle. The interpolation math can produce values for areas not within the triangle; they just don't make sense.

gl_FragCoord, being an interpolated value, could therefore generate values outside of the triangle. Since the grey triangle is aimed towards the viewer, the values from locations "above" the oncoming edge of the grey triangle will be closer than they should be. And since the edge of the grey triangle intersects the white triangle, values closer than its actual edge values will be considered closer than the white triangle

The normal way to counter this would be to use the centroid interpolation qualifier. However, the standard doesn't really allow this; even if you redeclared gl_FragCoord with the centroid qualifier, it won't have any effect:

The use of centroid does not further restrict this value to be inside the current primitive.

Also, as previously stated, depth-replacement in regular multisampled rendering destroys all of the per-sample depth information anyway. Every sample in a pixel would get the same depth value if your FS writes to the depth. That's not really what you wanted, even if you could do centroid interpolation of gl_FragCoord (which is probably why they don't allow it).

So if it is absolutely essential to do depth-replacement in a shader used for multisampling (and you should avoid this whenever possible), you will need to use per-sample shading. You can redeclare gl_FragCoord with sample to achieve this.