Odd effect with GLSL normals

Question

As a somewhat similar to a problem I had before and posted before, I'm trying to get normals to display correctly in my GLSL app. For the purposes of my explanation, I'm using the ninjaHead.obj model provided with RenderMonkey for testing purposes (you can grab it here). Now in the preview window in RenderMonkey, everything looks great: and the vertex and fragment code generated respectively is:

Vertex:

uniform vec4 view_position;

varying vec3 vNormal;
varying vec3 vViewVec;

void main(void)
{
  gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;

  // World-space lighting
  vNormal = gl_Normal;
  vViewVec = view_position.xyz - gl_Vertex.xyz;
}

Fragment:

uniform vec4 color;

varying vec3 vNormal;
varying vec3 vViewVec;

void main(void)
{
   float v = 0.5 * (1.0 + dot(normalize(vViewVec), vNormal));
   gl_FragColor =  v* color;
}

I based my GLSL code on this but I'm not quite getting the expected results...

My vertex shader code:

uniform mat4 P;
uniform mat4 modelRotationMatrix;
uniform mat4 modelScaleMatrix;
uniform mat4 modelTranslationMatrix;
uniform vec3 cameraPosition;

varying vec4 vNormal;
varying vec4 vViewVec;

void main()
{
vec4 pos = gl_ProjectionMatrix * P * modelTranslationMatrix * modelRotationMatrix * modelScaleMatrix * gl_Vertex;

gl_Position = pos;

gl_TexCoord[0] = gl_MultiTexCoord0;

gl_FrontColor = gl_Color;   

vec4 normal4 = vec4(gl_Normal.x,gl_Normal.y,gl_Normal.z,0);     

// World-space lighting
   vNormal = normal4*modelRotationMatrix;
   vec4 tempCameraPos = vec4(cameraPosition.x,cameraPosition.y,cameraPosition.z,0);     
   //vViewVec = cameraPosition.xyz - pos.xyz;
   vViewVec = tempCameraPos - pos;
}

My fragment shader code:

varying vec4 vNormal;
varying vec4 vViewVec;

void main()
{
  //gl_FragColor = gl_Color;
  float v = 0.5 * (1.0 + dot(normalize(vViewVec), vNormal));
gl_FragColor =  v * gl_Color;
}

However my render produces this...

Does anyone know what might be causing this and/or how to make it work?

EDIT In response to kvark's comments, here is the model rendered without any normal/lighting calculations to show all triangles being rendered.

And here is the model shading with the normals used for colors. I believe the problem has been found! Now the reason is why it is being rendered like this and how to solve it? Suggestions are welcome!

SOLUTION Well everyone the problem has been solved! Thanks to kvark for all his helpful insight that has definitely helped my programming practice but I'm afraid the answer comes from me being a MASSIVE tit... I had an error in the display() function of my code that set the glNormalPointer offset to a random value. It used to be this:

gl.glEnableClientState(GL.GL_NORMAL_ARRAY);
gl.glBindBuffer(GL.GL_ARRAY_BUFFER, getNormalsBufferObject());
gl.glNormalPointer(GL.GL_FLOAT, 0, getNormalsBufferObject());

But should have been this:

gl.glEnableClientState(GL.GL_NORMAL_ARRAY);
gl.glBindBuffer(GL.GL_ARRAY_BUFFER, getNormalsBufferObject());
gl.glNormalPointer(GL.GL_FLOAT, 0, 0);

So I guess this is a lesson. NEVER mindlessly Ctrl+C and Ctrl+V code to save time on a Friday afternoon AND... When you're sure the part of the code you're looking at is right, the problem is probably somewhere else!

Answer 1

You seem to be mixing GL versions a bit? You are passing the matrices manually via uniforms, but use fixed function to pass vertex attributes. Hm. Anyway...

---

I sincerely don't like what you're doing to your normals. Have a look:

vec4 normal4 = vec4(gl_Normal.x,gl_Normal.y,gl_Normal.z,0);     

vNormal = normal4*modelRotationMatrix;

A normal only stores directional data, why use a vec4 for it? I believe it's more elegant to just use just vec3. Furthermore, look what happens next- you multiply the normal by the 4x4 model rotation matrix... And additionally your normal's fourth cordinate is equal to 0, so it's not a correct vector in homogenous coordinates. I'm not sure that's the main problem here, but I wouldn't be surprised if that multiplication would give you rubbish.

The standard way to transform normals is to multiply a vec3 by the 3x3 submatrix of the model-view matrix (since you're only interested in the orientation, not the translation). Well, precisely, the "correctest" approach is to use the inverse transpose of that 3x3 submatrix (this gets important when you have scaling). In old OpenGL versions you had it precalculated as gl_NormalMatrix.

So instead of the above, you should use something like

// (...)
varying vec3 vNormal;
// (...)

mat3 normalMatrix = transpose(inverse(mat3(modelRotationMatrix)));
// or if you don't need scaling, this one should work too-
mat3 normalMatrix = mat3(modelRotationMatrix);

vNormal = gl_Normal*normalMatrix;

That's certainly one thing to fix in your code - I hope it solves your problem.

Answer 2

1. What is your P matrix? (I suppose it's a world->camera view transform).

2. vNormal = normal4*modelRotationMatrix; Why did you change the order of arguments? Doing that you are multiplying the normal by inversed rotation, what you don't really want. Use the standard order instead (modelRotationMatrix * normal4)

3. vViewVec = tempCameraPos - pos. This is entirely incorrect. pos is your vertex in the homogeneous clip-space, while tempCameraPos is in world space (I suppose). You need to have the result in the same space as your normal is (world space), so use world-space vertex position (modelTranslationMatrix * modelRotationMatrix * modelScaleMatrix * gl_Vertex) for this equation.