mouse picking in opengl using `gluUnProject`

Question

I have a scene that I am rendering couple of cubes in it with openGL (program structure is not using GLUT,it is in win32 program structure but I just draw cubes with glutSolidCube) now I want to select these cubes by mouse by picking. this is what I am doing: first when the user clicks the mouse button on the scene I get the mouse position and trying to find its coordinates in scene coordinates (templateSkeletons is a skeleton I created out of cubes, nothing more):

if (mouse.buttonPressed(Mouse::BUTTON_LEFT))
        {
            mouse.update();
            templateSkeletons[0].selectionMode = true;
            Vector3* points;
            points = GetOGLPos();
            templateSkeletons[0].setIntersectionPoints(points[0],points[1]);
        }else
            templateSkeletons[0].selectionMode = false;

this is the GerOGLPos function that I am retrieving the coordinates in the scene (notice that I have my own camrea and its own projection matrix but I am fetching projection matrix here in this function just by calling glGetDoublev (GL_PROJECTION_MATRIX, projmatrix); is this wrong and I should get my own camrea's projection matrix ? ) :

Vector3* GetOGLPos()
  {Vector3 pointsOnLine[2];
double mvmatrix[16];
double projmatrix[16];
int viewport[4];
double dX, dY, dZ, dClickY,zz;  
glGetIntegerv(GL_VIEWPORT, viewport);   
glGetDoublev (GL_MODELVIEW_MATRIX, mvmatrix);
glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);
dClickY = double (viewport[3] - mouse.yPos()); 
// OpenGL renders with (0,0) on bottom, mouse reports with (0,0) on top
//glReadPixels( mouse.xPos(), int(dClickY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &zz );
gluUnProject ((double) mouse.xPos(), dClickY, 0.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);
pointsOnLine[0] = Vector3( (float) dX, (float) dY, (float) dZ );
gluUnProject ((double) mouse.xPos(), dClickY, 1.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);
pointsOnLine[1]  = Vector3( (float) dX, (float) dY, (float) dZ );

    return pointsOnLine;
       }

now I suppose I have two points indicating my picking ray in the scene. now when I render the cubes I try to calculate the distance of the line created by the ray and the cube, and if it is smaller than a value I change the color of the cube to know that I picked it (jointsOfSkeleton indicates each cube that is created the skeleton nothing more, in here I am testing just the cube number 6 in array ):

 if(selectionMode)
        {

            distToLine  = Vector3::PointToLineDistance3D(rayPoints[0],rayPoints[1],Vector3::Vector3(jointsOfSkeleton[6].x,
                jointsOfSkeleton[6].y,jointsOfSkeleton[6].z));
            //distToLine = sqrt(distToLine);
            if(distToLine < 0.5)
                glColor3f(1.0,0.0,0.0);

            else 
                glColor3f(1.0,1.0,1.0);
        }

when I click on irrelevant positions on the window I see the colors of cube changes, it doesn't work right, I am watching the distances on the debugger and the distances doesn't look right. and this is function I used for finding Line-point distance:

static float PointToLineDistance3D(Vector3 a, Vector3 b, Vector3 point)
{   

    Vector3 lineDirection = b - a;
    float t = (Vector3::dot(point,lineDirection) - Vector3::dot(lineDirection,a))/(Vector3::dot(lineDirection,lineDirection));
    Vector3 direction;
    direction.x = a.x + (lineDirection.x *t) - point.x;
    direction.y = a.y + (lineDirection.y *t) - point.y;
    direction.z = a.z + (lineDirection.z *t) - point.z;

    float ShortestDistance = sqrtf((direction.x*direction.x)+(direction.y*direction.y)+(direction.z*direction.z));

    return ShortestDistance;

}

Answer 1

This is how I would write the PointToLineDistance3D:

static float PointToLineDistance3D(const Vector3 &a, const Vector3 &b, const Vector3 &point){   
    Vector3 lineDirection = Vector3::normalize(b - a), pointDirection = point - a;
    float t = Vector3::dot(pointDirection,lineDirection);
    Vector3 projection = a + (lineDirection * t);

   float ShortestDistance = (projection - point).length();
   return ShortestDistance;
}

I made the assumption that:

• the Vector3 class has a length method, with obvious meaning,
• there's a * operator to scale vectors,
• there's a normalize function as well which returns ... a normalized vector (this could also be made a method to avoid constructing an extra object).

The idea is to compute the projection of point on the ray and then compute the distance between the projection and point. As you can see the algorithm is slightly different from your implementation, most notably in the computation of t. Perhaps this is where your problem lies.

In order to test case the code I provided above, I wrote a small programme using it, which build a 3x3 wall of cubes on the XY plane, the centre of the wall being <0,0,0>. I managed to have it working without problem, even when moving the camera around. The only issue was with regard to the mouse coordinate system which goes top to bottom (ie. t_the Y mouse coord increases downward_), which is the opposit of the natural OpenGL Y axis. It requires the SDL library in order to compile and run.

#include <iostream>

#include <GL/gl.h>
#include <GL/glu.h>
#include <SDL/SDL.h>
#include <unistd.h>

#include <Vector3.h>

#define WIDTH 800
#define HEIGHT 600

GLuint box;
int highlight[2]; // position of the cube in the wall
const float cube_width = 5.0; 

Vector3 position(0,0,-25); // camera position

// build the cube display list
void setup_cube(){

  const float w = cube_width;

  float w0 = -w, h0 = -w, w1 = w, h1 = w;

  box = glGenLists(1);
  glNewList(box, GL_COMPILE);
    glBegin(GL_QUAD_STRIP);
      glVertex3f(w0, h1, w0);
      glVertex3f(w0, h0, w0 );
      glVertex3f(w1, h1, w0 );
      glVertex3f(w1, h0, w0 );
      glVertex3f(w1, h1, w1 );
      glVertex3f(w1, h0, w1 );
      glVertex3f(w0, h1, w1 );
      glVertex3f(w0, h0, w1 );
    glEnd();
    glBegin(GL_QUAD_STRIP);
      glVertex3f(w1, h1, w0 );
      glVertex3f(w1, h1, w1 );
      glVertex3f(w0, h1, w0 );
      glVertex3f(w0, h1, w1 );
      glVertex3f(w0, h0, w0 );
      glVertex3f(w0, h0, w1 );
      glVertex3f(w1, h0, w0 );
      glVertex3f(w1, h0, w1 );
    glEnd();
  glEndList();
}

void setup_scene(){

  float r = WIDTH / HEIGHT;

  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  glFrustum( -r, r, -1, 1, 1, 1024);

  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();

  glTranslatef(position[0],position[1],position[2]);

  glEnable(GL_CULL_FACE);
  glEnable(GL_DEPTH_TEST);
}

void draw_scene(){

  const float w = cube_width;
  int i = 0, j = 0;

  for (int i = -1; i < 2; i++) {
    for (int j = -1; j < 2; j++) {
      float x = w * 2 * i,  y = w * 2 * j;

      if (highlight[0] == i && highlight[1] == j)
         glColor3f(0.0, 1.0, 0.0);
      else 
         glColor3f(1.0, 0.0, 0.0);

      glPushMatrix ();
         glTranslatef(x,y,0);
         glCallList(box);
      glPopMatrix ();
    }
  }
}

void aim(float xm, float ym_){

  const float w = cube_width;
  float ym = HEIGHT - ym_;

  GLdouble model[16];
  GLdouble proj[16];
  GLint view[16];

  glGetDoublev(GL_MODELVIEW_MATRIX, model);
  glGetDoublev(GL_PROJECTION_MATRIX, proj);
  glGetIntegerv(GL_VIEWPORT, view);
  highlight[0] = -5;
  highlight[1] = -5;

  for (int i = -1; i < 2; i++) {
    for (int j = -1; j < 2; j++) {
      float x = w * 2 * i, y = w * 2 * j;
      double ox, oy, oz;
      Vector3 centre(x,y,0);
      gluUnProject(xm, ym, 0, model, proj, view, &ox, &oy, &oz);
      Vector3 p0(ox,oy,oz);
      gluUnProject(xm, ym, 1, model, proj, view, &ox, &oy, &oz);
      Vector3 p1(ox,oy,oz);
      float d = PointToLineDistance(p0,p1,centre);
      if (d < w) {
        highlight[0] = i;
        highlight[1] = j;
        return;
      }
    }
  }
}

int main(){

  SDL_Surface *screen;

  SDL_Init(SDL_INIT_VIDEO);

  SDL_GL_SetAttribute( SDL_GL_RED_SIZE, 5 );
  SDL_GL_SetAttribute( SDL_GL_GREEN_SIZE, 5 );
  SDL_GL_SetAttribute( SDL_GL_BLUE_SIZE, 5 );
  SDL_GL_SetAttribute( SDL_GL_DEPTH_SIZE, 16 );
  SDL_GL_SetAttribute( SDL_GL_DOUBLEBUFFER, 1 );

  if ( (screen=SDL_SetVideoMode( WIDTH, HEIGHT, 32, SDL_OPENGL )) == NULL ) {
    SDL_Quit();
    return -1;
  }

  setup_cube();

  while (1) {
    SDL_Event event;
    setup_scene();

    while(SDL_PollEvent(&event)){
      switch(event.type){
      case SDL_MOUSEMOTION:
        aim(event.motion.x, event.motion.y);
        break;
      case SDL_KEYDOWN: 
        {
          switch (event.key.keysym.sym){
          case SDLK_ESCAPE:
            SDL_Quit();
            exit(1);
          case SDLK_LEFT:
            position.add(Vector3(1,0,0));
            break;
          case SDLK_RIGHT:
            position.sub(Vector3(1,0,0));
            break;
          case SDLK_UP:
            position.add(Vector3(0,0,1));
            break;
          case SDLK_DOWN:
            position.sub(Vector3(0,0,1));
            break;
          case SDLK_PAGEDOWN:
            position.add(Vector3(0,1,0));
            break;
          case SDLK_PAGEUP:
            position.sub(Vector3(0,1,0));
            break;
          }
        }
      default:
        break;
      }
    }
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    draw_scene();
    SDL_GL_SwapBuffers();
    usleep(10);
  }

  return 0;
}

The source listed above displays correctly the cube aimed by the mouse pointer. It is possible to move around using the arrow and page up/down keys.