Striation in my shadows

Question

I'm casting shadows across a screenspace texture based on the values in another texture.

My "depth" texture, it's not really depth, just colour values that are sampled for heights, looks like this:

We can say that the red channel is my heightmap.

I draw this alongside a grass texture on a fullscreen quad with the following frag shader:

#version 400

layout(location=0) out vec4 frag_colour;

in vec2 texelCoords;

uniform sampler2D uTexture;
uniform sampler2D uTextureHeightmap;
uniform float uSunDistance = -10000000.0;
uniform float uSunInclination;
uniform float uSunAzimuth;
uniform float uQuality;

void main()
{
    vec4 c = texture(uTexture,texelCoords);

    vec2 textureD = textureSize(uTexture,0);
    float d = max(textureD.x,textureD.y);
    float aspectCorrection = textureD.x / textureD.y;

    vec3 sunPosition = vec3(textureD.x/2,textureD.y/2,0) + vec3(    uSunDistance*sin(uSunInclination)*cos(uSunAzimuth),
                                                                    uSunDistance*sin(uSunInclination)*sin(uSunAzimuth),
                                                                    uSunDistance*cos(uSunInclination)   );

    vec4 heights = texture(uTextureHeightmap, texelCoords);
    float height = max(max(heights.r,heights.g),heights.b);
    vec3 direction = normalize(vec3(texelCoords,height) - sunPosition);
    direction.y *= aspectCorrection;

    float sampleDistance = 0;

    float samples = d*uQuality;

    float stepSize = 1.0 / ((samples/d) * d);

    for(int i = 0; i < samples; i++)
    {
        sampleDistance += stepSize;

        vec3 newPoint = vec3(texelCoords,height) + direction * sampleDistance;
        if(newPoint.z > 1.0)
            break;

        vec4 h = texture(uTextureHeightmap,newPoint.xy);
        float base = h.r;
        float middle = h.g;
        float top = h.b;

        if(newPoint.z < base)
        {
            c *= 0.5;
            break;
        }
        if(newPoint.z >= middle && newPoint.z <= top)
        {
            c *= 0.5;
            break;
        }
    }

    frag_colour = c;
}

A sample of the output is:

The striation is not wanted. I can use this same method with sharper "edges" instead of smooth contours, and everything looks great. It's the gradients like this that cause problems.

Answer 1

To improve the quality of the algorithm, of course the number of samples can be increased. Buy a quality improvement can also be achieved, by a limitation of the stride distance.

The minimum meaningful distance of a step is determined by the resolution of the height map, because it makes no sense to test the same height of the height map twice.

float d = max(textureD.x,textureD.y);
float minStepSize = 1.0 / d;

The maximum meaningful distance is reached, if the distance, of the beam from the position on the ground to the sun position, reaches a height of 1.0.

The distance of one step is given, by dividing the maximum distance by the number of samples, but it should be at least the minimum distance given by the resolution of the height map:

float minStepSize = 1.0 / d;
float maxDist     = (1.0 - height) * length(direction.xy) / abs(direction.z);
float stepSize    = max( minStepSize, maxDist / samples );
float shadow      = 1.0;
vec3  startPoint  = vec3( texelCoords, height );
for ( float sampleDist = stepSize; sampleDist <= maxDist; sampleDist += stepSize )
{
    vec3 samplePoint   = startPoint + direction * sampleDist;
    vec4 sampleHeight  = texture( uTextureHeightmap, samplePoint.xy );
    if ( samplePoint.z < sampleHeight.r )
    {
        shadow *= 0.5;
        break;
    }
}
frag_colour = vec4( c.rgb * shadow, c.a );

For a soft shadow algorithm it has to be distinguished between full shadow and a shadow transition. For this the distance of the light beam, to to the height map has to be investigated.
If there is a large distance, then the fragment is in full shadow:

But in the case of a tiny distance, there is a shadow transition:

The indication of the strength of the shadow is the maximum distance, between the light beam and the height of the height map. This distance can be calculated by the maximum height difference of a sample point to the height map. A smooth shadow transition can be calculated by the GLSL function smoothstep.

vec3  startPoint  = vec3( texelCoords, height );
float maxHeight   = 0.0;
for ( float sampleDist = stepSize; sampleDist <= maxDist; sampleDist += stepSize )
{
    vec3 samplePoint   = startPoint + direction * sampleDist;
    vec4 sampleHeight  = texture( uTextureHeightmap, samplePoint.xy );
    maxHeight          = max(maxHeight, sampleHeight.r - samplePoint.z);
}
const float minShadow        = 0.5; 
const float transitionHeight = 0.05;
shadow    = smoothstep(1.0, minShadow, clamp(maxHeight / transitionHeight, 0.0, 1.0 ));
fragColor = vec4( height, height * shadow, height, 1.0 );

In the code above, the variable maxHeight contains the maximum distance, between the light beam and the height of the height map. If it is greater than 0.0 and less than transitionHeight, there is a shadow transition. If it is greater than transitionHeight, there is a full shadow. The strength of full shadow is set by minShadow.