Why does this glsl shader not respect the depth index?

Question

This is a vertex shader I'm currently working with:

attribute vec3 v_pos;
attribute vec4 v_color;
attribute vec2 v_uv;
attribute vec3 v_rotation; // [angle, x, y]

uniform mat4 modelview_mat;
uniform mat4 projection_mat;

varying vec4 frag_color;
varying vec2 uv_vec;
varying mat4 v_rotationMatrix;

void main (void) {

    float cos = cos(v_rotation[0]);
    float sin = sin(v_rotation[0]);

    mat2 trans_rotate = mat2(
      cos, -sin,
      sin, cos
    );

    vec2 rotated = trans_rotate * vec2(v_pos[0] - v_rotation[1], v_pos[1] - v_rotation[2]);
    gl_Position = projection_mat * modelview_mat * vec4(rotated[0] + v_rotation[1], rotated[1] + v_rotation[2], 1.0, 1.0);
    gl_Position[2] = 1.0 - v_pos[2] / 100.0; // Arbitrary maximum depth for this shader.
    frag_color = vec4(gl_Position[2], 0.0, 1.0, 1.0);  // <----------- !!
    uv_vec = v_uv;
}

and the fragment:

varying vec4 frag_color;
varying vec2 uv_vec;

uniform sampler2D tex;

void main (void){
    vec4 color = texture2D(tex, uv_vec) * frag_color;
    gl_FragColor = color;
}

Notice how I'm manually setting the Z index of the gl_Position variable to be a bound value in the range 0.0 -> 1.0 (the upper bound is done in code; safe to say, not vertex has a z value < 0 or > 100).

It works... mostly. The problem is that when I render it, I get this:

That's not the correct depth sorting for these elements, which have z value respectively, of 15, 50 and 80, as you can see from the red value for each sprite.

The correct order to render in would be blue -> top, purple middle, and pink bottom; but instead these sprites are being rendered in render order.

ie. They are being drawn via:

glDrawArrays() <--- Pink, first geometry batch
glDrawArrays() <--- Blue, second geometry batch
glDrawArrays() <--- Purple, thirds geometry batch

What's going on?

Surely it irrelevant how many times I call gl draw functions before flushing; the depth testing should sort this all out right?

Do you have to manually invoke depth testing inside the fragment shader somehow?

Answer 1

You say you're normalizing the output Z value into the range: 0.0 - 1.0?

It should really be the range: -W - +W. Given an orthographic projection, this means the clip-space Z should range from: -1.0 - +1.0. You are only using half of your depth range, which reduces the resolving capability of the depth buffer significantly.

To make matters worse (and I am pretty sure this is where your actual problem comes from) it looks like you are inverting your depth buffer by giving the farthest points a value of 0.0 and the nearest points 1.0. In actuality, -1.0 corresponds to the near plane and 1.0 corresponds to the far plane in OpenGL.

gl_Position[2] = 1.0 - v_pos[2] / 100.0;
~~~~~~~~~~~~~~
// This needs to be changed, your depth is completely backwards.


gl_Position.z = 2.0 * (v_pos.z / 100.0) - 1.0;
~~~~~~~~~~~~~
// This should fix both the direction, and use the full depth range.

However, it is worth mentioning that now the value of gl_Position[2] or gl_Position.z ranges from -1.0 to 1.0 which means it cannot be used as a visible color without some scaling and biasing:

frag_color = vec4 (gl_Position.z * 0.5 + 0.5, 0.0, 1.0, 1.0);  // <----------- !!

On a final note, I have been discussing Normalized Device Coordinates this entire time, not window coordinates. In window coordinates the default depth range is 0.0 = near, 1.0 = far; this may have been the source of some confusion. Understand that window coordinates (gl_FragCoord) are not pertinent to the calculations in the vertex shader.

You can use this in your fragment shader to test if your depth range is setup correctly:

vec4 color = texture2D(tex, uv_vec) * vec4 (gl_FragCoord.z, frag_color.yzw);

It should produce the same results as:

vec4 color = texture2D(tex, uv_vec) * frag_color;