LIBGDX 3D: unable to accurately determine which mesh triangle is hit by a camera ray

Question

I am learning to use LIBGDX to create a minecraft alike world, created out of textured blocks (cubes). I have the 3D world rendered and the player movement is working flawlessly (the result and movement experience thru the 3D world feels as expected).

I am now aiming to detect which side/face of a block is pointed at by the camera from a firstperson perspective. Using the Intersector.intersectRayBounds I managed to determine which block is pointed at. I draw a boundingbox around the found block for visual result.

The next step is to visually mark the exact side/face of the block (cube). I am trying to achieve this by using the Intersector.intersectRayTriangles after iterating thru the list of meshparts of the found block and retrieve the indices/vertices of the mesh.

The current result is that I am able to mark the side/face of the cube which is pointed at, however, it is inaccurate. Only when "pointing at" the block in a specific corner it will result in a positive intersection - as if I am only able to test a subset of triangles rather than all.

Code example to determine which block(cube) is pointed at:

Ray ray = camera.getPickRay(1280/2, 720/2+BLOCK_SIZE);//, 0, 0, 1280, 720);
float distance = -1f;
float distance2;
Vector3 position = new Vector3();
Vector3 intersection = new Vector3();
BoundingBox bb = new BoundingBox();
for (BlockBase3D block:mBlocksArray) {
  bb.set(block.getBoundingBox());
  bb.mul(block.getModelInstance().transform);
  position.set(block.getCenterPosition());
  distance2 = ray.origin.dst2(position);
  if (distance >= 0f && distance2 > distance) continue;
  if (Intersector.intersectRayBounds(ray, bb, intersection)) {
    mBlockPointed = block;
    distance = distance2;
  }
}
mBlockPointed.setIsShowingBoundingBox(true);

This part of the code results in a found block mBlockPointed.

With the next code part I try to determine the exact side/face of the block pointed at:

if (mBlockPointed != null){        
  NodePart np = null;
  MeshPart mp = null;
  float[] meshPartVertices = {};
  short[] meshPartIndices = {};

  Mesh mesh;
  int vertexSize;
  ray = camera.getPickRay(1280/2, 720/2+BLOCK_SIZE);

  for (int j=0; j<mBlockPointed.getModelInstance().nodes.size; j++){
    for (int i = 0; i < mBlockPointed.getModelInstance().nodes.get(j).parts.size; i++) {
      np = mBlockPointed.getModelInstance().nodes.get(j).parts.get(i);
      mp = np.meshPart;
      mesh = mp.mesh.copy(false);
      mesh.transform(mBlockPointed.getModelInstance().transform);
      meshPartIndices = new short[mp.size];
      mesh.getIndices(mp.offset, mp.size, meshPartIndices, 0);
      vertexSize = mesh.getVertexSize() / 4;
      meshPartVertices = new float[mesh.getNumVertices() * vertexSize];
      mesh.getVertices(meshPartVertices);

      if (Intersector.intersectRayTriangles(ray, meshPartVertices, meshPartIndices, vertexSize, null)) {
        np.material.set(mSelectionMaterial);
        //break;
      }                        
    }
  }

My thought behind this codepart are:

• Iterate over all nodes of the modelinstance (currently only one node involved)
• Iterate over all nodeparts of the node(s) of the modelinstance (6 for each block side/face)
• Get the mesh of the node and transform the mesh according the found block pointed at (to get the world position, rotation and scaling).
• get indices based upon the offset in the indices buffer for the current meshpart
• get all vertices of the mesh, based upon the number of vertices and the size of the vertex used
• perform a intersect triangles test for the given ray and the indices/vertices for the current mesh part

The result is that when I move the (perspective-) camera around I get a positive result only when the camera points to one of the corner areas of the blockside. I am unable to think of the cause of this, which if I was able to, could set me off in the direction to solve this.

EDIT - working code after using ModelInstance.Renderer.worldTransform in stead of ModelInstance.transform

Also updated my code with some inline comments and optimization.

public void detectObjectPointed(int screenX, int screenY, PerspectiveCamera camera){
        Ray ray = camera.getPickRay(1280/2, 720/2+BLOCK_SIZE);
        float distance = -1f;
        float distance2;

        // if previous block found, restore original materials
        if (mBlockPointed != null){
            for (int i = 0; i < mBlockPointed.getModelInstance().nodes.get(0).parts.size; i++) {
                mBlockPointed.getModelInstance().nodes.get(0).parts.get(i).material.set(mOriginalMaterials.get(i));
            }
            mBlockPointed.setIsShowingBoundingBox(false);
            mBlockPointed = null;
        }

        // attempt to find block pointing at by camera ray
        for (BlockBase3D block:mBlocksArray) {
            mBlockPointedBoundingBox.set(block.getBoundingBox());
            mBlockPointedBoundingBox.mul(block.getModelInstance().transform);
            mBlockPointedPosition.set(block.getCenterPosition().cpy());
            distance2 = ray.origin.dst2(mBlockPointedPosition);
            if (distance >= 0f && distance2 > distance) continue;
            if (Intersector.intersectRayBounds(ray, mBlockPointedBoundingBox, mBlockPointedIntersection)) {
                mBlockPointed = block;
                distance = distance2;
            }
        }

        // if block pointed at is found
        if (mBlockPointed != null){
            // draw the boundingbox (wireframe) to visually mark the block pointed at
            mBlockPointed.setIsShowingBoundingBox(true);

            // get the mesh of the block pointed at and populate the vertices array; assumption made we have 1 mesh only in the model
            float[] meshPartVertices = {};
            short[] meshPartIndices = {};
            int vertexSize;
            // get the worldtransform matrix of the renderable from the ModelInstance
            Matrix4 m4 = mBlockPointed.getModelInstance().getRenderable(new Renderable()).worldTransform;
            mBlockPointedMesh = mBlockPointed.getModel().meshes.get(0).copy(false);
            // transform the vertices of the mesh to match world position/rotation/scaling
            mBlockPointedMesh.transform(m4);
            vertexSize = mBlockPointedMesh.getVertexSize() / 4; // a float is 4 bytes, divide by four to get the number of floats
            meshPartVertices = new float[mBlockPointedMesh.getNumVertices() * vertexSize];
            mBlockPointedMesh.getVertices(meshPartVertices);

            // clear the backup of original materials
            mOriginalMaterials.clear();
            // assume we have one model node only and loop over the nodeparts (expected is 6 nodeparts, 1 for each block/cube side)
            for (int i = 0; i < mBlockPointed.getModelInstance().nodes.get(0).parts.size; i++) {
                // get a nodepart and populate the indices array
                mBlockPointedNodePart = mBlockPointed.getModelInstance().nodes.get(0).parts.get(i);
                meshPartIndices = new short[mBlockPointedNodePart.meshPart.size];
                mBlockPointedMesh.getIndices(mBlockPointedNodePart.meshPart.offset, mBlockPointedNodePart.meshPart.size, meshPartIndices, 0);

                // backup the original material for this nodepart
                mOriginalMaterials.add(i, mBlockPointedNodePart.material.copy());

                // check if the ray intersects with one or more of the triangles for this nodepart
                if (Intersector.intersectRayTriangles(ray, meshPartVertices, meshPartIndices, vertexSize, null)) {
                    // intersection detected, visually mark the nodepart by setting a selection material
                    mBlockPointedNodePart.material.set(mSelectionMaterial);
                }
            }
        }