OpenGL 3D Picking

Question

I read many sample code about opengl picking. Nearly all of them use gluPerspective funcion for projection.I'm using glOrtho instead of gluPerspective function.

And my Selection function is as below(DrawBuffer is my paint code):

void Selection( int x, int y )
{
    GLuint buffer[512];
    GLint hits;

    GLint viewport[4];
    glGetIntegerv(GL_VIEWPORT, viewport);
    glSelectBuffer(512, buffer);

    (void)glRenderMode(GL_SELECT);

    glInitNames();
    glPushName(0);

    glMatrixMode(GL_PROJECTION);
    glPushMatrix();
    glLoadIdentity();

    GLdouble w = (double)m_ClientRect.Width();
    GLdouble h = (double)m_ClientRect.Height();
    gluPickMatrix((GLdouble)x, (GLdouble)(viewport[3] - y), 500, 500, viewport);
    glOrtho(-w / 2, w / 2, -h / 2, h / 2, -1000000.0, 100000.0);
    glMatrixMode(GL_MODELVIEW);
    DrawBuffer();
    glMatrixMode(GL_PROJECTION);
    glPopMatrix();
    glMatrixMode(GL_MODELVIEW);
    hits = glRenderMode(GL_RENDER);

    if (hits > 0)
    {
        TRACE(_T("%d %d %d %d %d\n"), hits, buffer[0], buffer[1], buffer[2], buffer[3]);
    }
}

But it doesn't work, I can't figure out the reason? Another problem is: When I using glDrawArrays function to draw many lines, how can I call glLoadName to flag each of them?

My raytracer algorithm is as follow:

void CGraphicView::KDSearch( PICKING_VERTEX *root, CRay *pRay, PICKING_VERTEX **found, double *dCurSplit )
{
    if (NULL == root)
    {
        return;
    }
    SearchNode(root, m_pRay, m_globaltMin, m_globaltMax, found, dCurSplit);
}

void CGraphicView::SearchNode( PICKING_VERTEX *node, CRay *pRay, double tmin, double tmax, PICKING_VERTEX **found, double *dCurSplit )
{
    if (NULL == node)
    {
        return;
    }
    if (node->bLeaf)
    {
        SearchLeaf(node, pRay, tmin, tmax, found, dCurSplit);
    }
    else
    {
        SearchSplit(node, pRay, tmin, tmax, found, dCurSplit);
    }
}

void CGraphicView::SearchSplit( PICKING_VERTEX *split, CRay *pRay, double tmin, double tmax, PICKING_VERTEX **found, double *dCurSplit )
{
    if (NULL == split)
    {
        return;
    }
    int axis = split->axis;
    double thit = pRay->GetSplit(axis, split->coor[axis]);

    Point3D pSrc(split->coor[0], split->coor[1], split->coor[2]);
    double scale = m_pCam->GetScale();
    double disP2L = DistanceP2L(pSrc, m_RayStart, m_RayEnd);
    if (disP2L * scale < MAX_DISTANCE && thit < *dCurSplit)
    {
        *found = split;
        *dCurSplit = thit;
    }

    PICKING_VERTEX *first = NULL, *second = NULL;
    if (IS_EQUAL_FLOAT(pRay->m_direction[axis], 0.0))
    {
        first = (pRay->m_origin[axis] < split->coor[axis]) ? split->left : split->right;
    }
    else
    {
        first = (pRay->m_direction[axis] > 0.0) ? split->left: split->right;
        second = (pRay->m_direction[axis] < 0.0) ? split->left : split->right;
    }

    if ((thit >= tmax || thit < 0))
    {
        SearchNode(first, pRay, tmin, tmax, found, dCurSplit);
    }
    else if (thit <= tmin)
    {
        SearchNode(second, pRay, tmin, tmax, found, dCurSplit);
    }
    else
    {
        SearchNode(first, pRay, tmin, thit, found, dCurSplit);
    }
}

void CGraphicView::SearchLeaf( PICKING_VERTEX *leaf, CRay *pRay, double tmin, double tmax, PICKING_VERTEX **found, double *dCurSplit )
{
    if (NULL == leaf)
    {
        return;
    }

    int axis = leaf->axis;
    double thit = pRay->GetSplit(axis, leaf->coor[axis]);

    Point3D pSrc(leaf->coor[0], leaf->coor[1], leaf->coor[2]);
    double scale = m_pCam->GetScale();
    double disP2L = DistanceP2L(pSrc, m_RayStart, m_RayEnd);
    if (disP2L * scale < MAX_DISTANCE && thit < *dCurSplit)
    {
        *found = leaf;
        *dCurSplit = thit;
    }

    ContinueSearch(leaf, pRay, tmin, tmax, found, dCurSplit);
}

void CGraphicView::ContinueSearch( PICKING_VERTEX *leaf, CRay *pRay, double tmin, double tmax, PICKING_VERTEX **found, double *dCurSplit )
{
    if (IS_EQUAL_FLOAT(tmax, m_globaltMax))
    {
        return;
    }
    else
    {
        tmin = tmax;
        tmax = m_globaltMax;
        SearchNode(m_root, pRay, tmin, tmax, found, dCurSplit);
    }
}

Answer 1

Well, from memory one way to debug hits == 0 was to use exactly the same pick matrix for a normal render, or in this case just comment out the glRenderMode calls. If you don't see anything drawn, then no part of your scene intersects the pick area and the selection code is just doing what you told it to.

However, datenwolf is right and you really should avoid OpenGL selection mode. It's horrible.

A fairly simple way to implement picking that doesn't require raycasting or Kd-trees is to draw each object in a different color. Assuming every object has a unique identifier number (which you'd need for glLoadName anyway), convert it into an 3 byte RGB color value. Draw the scene into the back buffer only, then read the pixel under the mouse coordinates. That RGB value will be the identifier of the frontmost object.

Answer 2

When I using glDrawArrays function to draw many lines, how can I call glLoadName to flag each of them?

You can't. And frankly: You should not use OpenGL selection mode in the first place! It's slow, no current driver supports it (you'll always drop back into software emulation mode with it), it doesn't work (well) with shaders and is cumbersome to use.

A far better alternative is to either backproject selection rays into the scene or (if modern OpenGL is used) to use a transform feedback buffer applied in bounding boxes (or other kind of bounding volume) to sort geometry into a screen space Kd-tree from where you can quickly select what's been clicked onto.