Parallax mapping works in only one direction

Question

I've implemented multiple variations of Steep Parallax, Relief and Parallax Occlusion mappings, and they all have a bug of only working correctly in one direction. Which leads me to believe the problem is one of the values being calculated outside of Parallax mapping. I cannot figure out what is wrong.

Here's a cube with Parallax mapping from different angles, as you can see Parallax effect is correct on only one side of the cube:

Here is the GLSL code that is used:

Vertex shader:

layout(location = 0) in vec3 vertexPosition;
layout(location = 1) in vec3 vertexNormal;
layout(location = 2) in vec2 textureCoord;
layout(location = 3) in vec3 vertexTangent;
layout(location = 4) in vec3 vertexBitangent;

out vec3 eyeVec;
out vec2 texCoord;

uniform vec3 cameraPosVec;  // Camera's position
uniform mat4 modelMat;      // Model matrix

void main(void)
{       
    texCoord = textureCoord;

    fragPos = vec3(modelMat * vec4(vertexPosition, 1.0));

    // Compute TBN matrix
    mat3 normalMatrix = transpose(inverse(mat3(modelMat)));
    TBN = mat3(normalMatrix * vertexTangent, 
               normalMatrix * vertexBitangent, 
               normalMatrix * vertexNormal);

    eyeVec = TBN * (fragPos - cameraPosVec);
}

Fragment shader:

layout(location = 1) out vec4 diffuseBuffer;

// Variables from vertex shader
in vec3 eyeVec;
in vec3 fragPos;
in vec2 texCoord;

uniform sampler2D diffuseTexture;
uniform sampler2D heightTexture;

vec2 parallaxOcclusionMapping(vec2 p_texCoords, vec3 p_viewDir)
{       
    // number of depth layers
    float numLayers = 50;

    // calculate the size of each layer
    float layerDepth = 1.0 / numLayers;
    // depth of current layer
    float currentLayerDepth = 0.0;
    // the amount to shift the texture coordinates per layer (from vector P)
    vec2 P = p_viewDir.xy / p_viewDir.z * 0.025;
    return p_viewDir.xy / p_viewDir.z;
    vec2 deltaTexCoords = P / numLayers;

    // get initial values
    vec2  currentTexCoords     = p_texCoords;
    float currentDepthMapValue = texture(heightTexture, currentTexCoords).r;
    float previousDepth = currentDepthMapValue;

    while(currentLayerDepth < currentDepthMapValue)
    {
        // shift texture coordinates along direction of P
        currentTexCoords -= deltaTexCoords;
        // get depthmap value at current texture coordinates
        currentDepthMapValue = texture(heightTexture, currentTexCoords).r;  

        previousDepth = currentDepthMapValue;
        // get depth of next layer
        currentLayerDepth += layerDepth;  
    }

    // -- parallax occlusion mapping interpolation from here on
    // get texture coordinates before collision (reverse operations)
    vec2 prevTexCoords = currentTexCoords + deltaTexCoords;

    // get depth after and before collision for linear interpolation
    float afterDepth  = currentDepthMapValue - currentLayerDepth;
    float beforeDepth = texture(heightTexture, prevTexCoords).r - currentLayerDepth + layerDepth;

    // interpolation of texture coordinates
    float weight = afterDepth / (afterDepth - beforeDepth);
    vec2 finalTexCoords = prevTexCoords * weight + currentTexCoords * (1.0 - weight);

    return finalTexCoords;
}

void main(void)
{   
    vec2 newCoords = parallaxOcclusionMapping(texCoord, eyeVec);

    // Get diffuse color
    vec4 diffuse = texture(diffuseTexture, newCoords).rgba;

    // Write diffuse color to the diffuse buffer
    diffuseBuffer = diffuse;
}

The code is literally copy-pasted from this tutorial

P.S. Inverting different values (x,y,z) of eyeVar vector (after transforming it with TBN matrix), changes the direction at which parallax works. For instance, inverting the X component, makes it parallax work on polygons facing upwards:

which suggests that there might be a problem with tangents/bitangents or the TBN matrix, but I have not found anything. Also, rotation the object with a model matrix does not affect the direction at which the parallax effect works.

EDIT:

I have fixed the issue by changing how I calculate tangents/bitangents, moving the viewDirection calculation to the fragment shader, and by not doing the matrix inverse for TBN for just this one calculation. See the newly added shader code here:

Vertex shader:

out vec3 TanViewPos;
out vec3 TanFragPos;

void main(void)
{       
    vec3 T = normalize(mat3(modelMat) * vertexTangent);
    vec3 B = normalize(mat3(modelMat) * vertexBitangent);
    vec3 N = normalize(mat3(modelMat) * vertexNormal);
    TBN = transpose(inverse(mat3(T, B, N)));

    mat3 TBN2 = transpose((mat3(T, B, N)));
    TanViewPos = TBN2 * cameraPosVec;
    TanFragPos = TBN2 * fragPos;
}

Fragment shader:

in vec3 TanViewPos;
in vec3 TanFragPos;

void main(void)
{       
    vec3 viewDir = normalize(TanViewPos - TanFragPos);
    vec2 newCoords = parallaxOcclusionMapping(texCoord, viewDir);
}

Regarding the tangent/bitangent calculation, I previously used ASSIMP to generate them for me. Now I've written code to manually calculate tangents and bitangents, however my new code generates "flat/hard" tangents (i.e. same tangent/bitangent for all 3 vertices of a triangle), instead of smooth ones. See the difference between ASSIMP (smooth tangents) and my (flat tangents) implementations:

The parallax mapping now works in all directions:

However, this now introduces another problem, which makes all the rounded objects have flat shading (after performing normal mapping):

Is this problem specific to my engine (i.e. a bug somewhere) or is this a common issue that other engines somehow deal with?

Answer 1

All the comments, your solution and edit missed the mark. While you've replaced the shader code with correct code, your work on smoothing tangents is not actually what fixed it. And the tangent/bitangent code from your solution is incorrect.

However, as I understand it, you're incorrect in several different places-

1. Starting with the code here mat3 normalMatrix = transpose(inverse(mat3(modelMat))); is actually called the inverse transpose, and it's used for lighting using non-uniform scaling objects.

Presumably your modelMat is a homogeneous transformation from object space to world space. Taking the inverse of it will transform from world space to object space - and because you've stripped the translation information by casting to a mat3, it will not 'un-translate' anything either. The transpose of that is called the inverse transpose, which effectively ONLY un-scales an object or vector, preserving ONLY the rotation information . You then multiplied the vertex tangent, bitangent and normal by this inverse scaling matrix to form an orthogonal basis of vectors (which previously were unit length) un-scaled by the same factor as your model- and still in tangent space. So your TBN at this stage is effectively some sort of strange rotated object-space to tangent-space transformation matrix which will also unscale a vector.

Had it been correct, you wouldn't actually need to use the inverse transpose for your lighting with the example objects because they're uniformly scaled.

2. The way you're using it was incorrect. eyeVec = TBN * (fragPos - cameraPosVec); First (assuming you're using thephong model), you were incorrectly taking the the vertex position (fragPos) minus the camera position to use as your view vector, which is reversed in direction from the phong model. You then multiplied it by your world to tangent matrix TBN, effectively transforming the world space vec3 into a tangent space vec3.

3. The code is somewhat correct in that the shader is using and creating the TBN matrix correctly. Multiplying each tangent space vector by the object to world transformation is correct because the way you attempted to fix it appears to be calculated in object space, and they represent the basis for a transformation to tangent space. Had you not used the inverse tangent again, it would have been even more correct. Taking the inverse transpose using the 3x3, again, will make the matrix into the exact same transformation that also unscales any 3D direction vector centered at the origin. This is why it didn't produce any noticeable effects, since all of your tangent, bitangent and normal vectors were uniform scale to begin with (unit length) the change is very minor and everything uses the same scale, so it's practically unnoticeable.

You then proceed to correctly use the regular transpose on the TBN matrix - Which is made possible because you orthonormalized your tangent and bitangent vectors in the corrected calculations from your response.

4. The most noticeable problem with your solution is that your CPU code indexes well past the bounds of every array multiple times, and is also incorrect. By walking the array 3 at a time, the last 2 indices will slide right out of bounds (you can try the % operator to wrap the last two)- AND because you're using the assignment operator here

m_tangents[i + 0] = Math::normalize(tangent - n0 * Math::dot(n0, tangent)); m_tangents[i + 1] = Math::normalize(tangent - n1 * Math::dot(n1, tangent)); m_tangents[i + 2] = Math::normalize(tangent - n2 * Math::dot(n2, tangent));

m_bitangents[i + 0] = Math::normalize(bitangent - n0 * Math::dot(n0, bitangent));
m_bitangents[i + 1] = Math::normalize(bitangent - n1 * Math::dot(n1, bitangent));
m_bitangents[i + 2] = Math::normalize(bitangent - n2 * Math::dot(n2, bitangent));

you're overwriting every tangent and bitangent vector every time you calculate. the only index being used is the [i + 0]. The math you're using is intended to be used on faces of triangles, so you should be iterating over the triangles instead. Additionally, this code assumes there are no duplicate vertices in this list- while simultaneously assuming that any 3 consecutive vertices form a valid triangle (which i'm not even sure is possible, certainly not without repeating verties but who knows). Essentially, you should be doing all of this by iterating over a list of triangles made up of indices into your vertex array, and you should be using the += operator to sum the tangents, then normalizing it to get the average for all faces.

A side note: In regards to your comments, perhaps you understood tangent space but the caulculations were certainly not correct, and parallax occlusion mapping does not work the same way with spherical UV mapping, because of the occlusion parameters you're using. Additionally, it might not have looked right for you if you were using the same planar mapping technique as you do on these flat objects (you would use spherical mapping instead, but you would not notice any occlusion)

Answer 2

The main underlying problem was with my tangents/bitangents. I was always relying on ASSIMP to calculate them for me, and it always worked fine (for normal mapping for instance) until now. The solution was to calculate them myself, using some smoothing process:

for(decltype(m_positions.size()) i = 0, size = m_positions.size(); i < size; i++)
{
    // Get vertex positions of the polygon
    const Math::Vec3f &v0 = m_positions[i + 0];
    const Math::Vec3f &v1 = m_positions[i + 1];
    const Math::Vec3f &v2 = m_positions[i + 2];

    // Get texture coordinates of the polygon
    const Math::Vec2f &uv0 = m_texCoords[i + 0];
    const Math::Vec2f &uv1 = m_texCoords[i + 1];
    const Math::Vec2f &uv2 = m_texCoords[i + 2];

    // Get normals of the polygon
    const Math::Vec3f &n0 = m_normals[i + 0];
    const Math::Vec3f &n1 = m_normals[i + 1];
    const Math::Vec3f &n2 = m_normals[i + 2];

    // Calculate position difference
    Math::Vec3f deltaPos1 = v1 - v0;
    Math::Vec3f deltaPos2 = v2 - v0;

    // Calculate texture coordinate difference
    Math::Vec2f deltaUV1 = uv1 - uv0;
    Math::Vec2f deltaUV2 = uv2 - uv0;

    // Calculate tangent and bitangent
    float r = 1.0f / (deltaUV1.x * deltaUV2.y - deltaUV1.y * deltaUV2.x);
    Math::Vec3f tangent = (deltaPos1 * deltaUV2.y - deltaPos2 * deltaUV1.y) * r;
    Math::Vec3f bitangent = (deltaPos2 * deltaUV1.x - deltaPos1 * deltaUV2.x) * r;

    // Orthogonalize using Gram–Schmidt process, to make tangents and bitangents smooth based on normal
    m_tangents[i + 0] = Math::normalize(tangent - n0 * Math::dot(n0, tangent));
    m_tangents[i + 1] = Math::normalize(tangent - n1 * Math::dot(n1, tangent));
    m_tangents[i + 2] = Math::normalize(tangent - n2 * Math::dot(n2, tangent));

    m_bitangents[i + 0] = Math::normalize(bitangent - n0 * Math::dot(n0, bitangent));
    m_bitangents[i + 1] = Math::normalize(bitangent - n1 * Math::dot(n1, bitangent));
    m_bitangents[i + 2] = Math::normalize(bitangent - n2 * Math::dot(n2, bitangent));
}