Issues drawing ray cast as line on click

Question

I've been working on converting screen to world coordinate for the last week or so (previous but unrelated question: Depth Component of Converting from Window -> World Coordinates).

I've made a lot of progress since that post in simplifying my code but now I am taking the ray cast approach for object selection, and more specifically visualizing the ray by drawing a line first (which is getting drawn but not with the correct coordinates).

(these 4 lines show what happens when clicking on each corner of the square + the camera has been rotated to view the lines better).

The code:

Vertex shader:

#version 330

layout(location = 0) in vec4 position;
layout(location = 1) in vec4 rcolor;

smooth out vec4 theColor;

uniform vec4 color;
uniform mat4 pv;

void main() {
  gl_Position = pv * position;
  theColor = color;
}

^ where pv is the combination of the projection matrix & view matrix. Relevant code from Camera class:

Camera::Camera()
{
  camPos = glm::vec3(0.0f, 5.0f, 1.0f);
  camLook = glm::vec3(0.0f, 0.0f, 0.0f);

  fovy = 90.0f;
  aspect = 1.0f;
  near = 0.1f;
  far = 100.0f;
}

glm::mat4 Camera::projectionMatrix()
{
  return glm::perspective(fovy, aspect, near, far);
}

glm::mat4 Camera::viewMatrix()
{
  return glm::lookAt(
    camPos,
    camLook,
    glm::vec3(0, 1, 0)
  );
}

glm::mat4 Camera::projectionViewMatrix()
{
  return projectionMatrix() * viewMatrix();
}

On click I initialize a Ray object:

void initializeRay(int x, int y)
{
  Ray rayObj(
    camera.getWorldNear(x, y),
    camera.getWorldFar(x, y)
  );
  rays.push_back(rayObj);
}

(relevant functions from the Camera class) - probably where the issue is

glm::vec3 Camera::getWorldNear(int x, int y)
{
  GLint viewport[4];
  glGetIntegerv(GL_VIEWPORT, viewport);

  glm::vec3 worldNear = glm::unProject(
    glm::vec3(x, viewport[3] - y, near),
    viewMatrix(),
    projectionMatrix(),
    glm::vec4(0.0f, 0.0f, viewport[2], viewport[3])
  );

  printf("near: (%f, %f, %f)\n", worldNear.x, worldNear.y, worldNear.z);

  return worldNear;
}

glm::vec3 Camera::getWorldFar(int x, int y)
{
  GLint viewport[4];
  glGetIntegerv(GL_VIEWPORT, viewport);

  glm::vec3 worldFar = glm::unProject(
    glm::vec3(x, viewport[3] - y, far),
    viewMatrix(),
    projectionMatrix(),
    glm::vec4(0.0f, 0.0f, viewport[2], viewport[3])
  );

  printf("far: (%f, %f, %f)\n", worldFar.x, worldFar.y, worldFar.z);

  return worldFar;
}

and construct the object....

Ray::Ray(glm::vec3 worldNear, glm::vec3 worldFar)
{
  float temp[] = {
    worldNear.x, worldNear.y, worldNear.z,
    worldFar.x, worldFar.y, worldFar.z,
  };

  vertexData.resize(6);
  for (x = 0; x < 6; x++)
    vertexData[x] = temp[x];

  glGenBuffers(1, &vbo);
  glBindBuffer(GL_ARRAY_BUFFER, vbo);
  glBufferData(GL_ARRAY_BUFFER, vertexData.size() * sizeof(float), &vertexData[0], GL_STATIC_DRAW);
  glBindBuffer(GL_ARRAY_BUFFER, 0);
}

And each time the display function is called, all rays are drawn:

void drawRays()
{
  for (int x = 0; x < rays.size(); x++) {
    glEnableVertexAttribArray(0);
    glBindBuffer(GL_ARRAY_BUFFER, rays[x].vbo);

    glVertexAttribPointer(
      0,
      2,
      GL_FLOAT,
      GL_FALSE,
      0,
      (void*)0
    );

    glUniform4f(colorUniform, 1.0f, 0.0f, 0.0f, 0.0f);

    glDrawArrays(GL_LINES, 0, 2);

    glDisableVertexAttribArray(0);
  }
}

My initial thought was that I was using the glm::unProject function in the wrong way, however I also tried changing the start and end position of the ray to the camPos & camLook (camera position & camera look at position) without any luck (again the rays seemed to all be clustered at the top of the square).

Edit:

I also made a function to get the direction of the ray, but it remains unused in my code at the moment:

glm::vec3 Camera::getRay(int x, int y)
{
  glm::vec3 worldNear = glm::unProject(
    glm::vec3(x, screenHeight - y, near),
    viewMatrix(),
    projectionMatrix(),
    glm::vec4(0.0f, 0.0f, screenWidth, screenHeight)
  );

  glm::vec3 worldFar = glm::unProject(
    glm::vec3(x, screenHeight - y, far),
    viewMatrix(),
    projectionMatrix(),
    glm::vec4(0.0f, 0.0f, screenWidth, screenHeight)
  );

  glm::vec3 direction = worldFar - worldNear;
  direction - glm::normalize(direction);

  printf("(%f, %f, %f)\n", direction.x, direction.y, direction.z);
  return direction;
}

Answer 1

To be absolutely clear, where are these rays you are visualizing supposed to originate? I imagine you are clicking some arbitrary location in window-space and want the ray to start there?

glm::UnProject (...) wants coordinates in window-space, but you are actually supplying near and far defined in view-space right now. It is easy to forget that transformation into window-space is defined by your viewport (NDC_x,y -> Win_x,y) and also depth range (NDC_z -> Win_z).

Your near and far plane positions go through multiple transformations:

   It starts out:

      near = 0.1 and far = 100.0 (view-space)

   After projection and perspective division:

      near = -1.0 and far = 1.0 (NDC space)

   Finally, after viewport transformation^*:

      near = 0.0 and far = 1.0 (window-space)

          ^{*The default depth range GL uses is [0.0, 1.0]}.

This means that after projection and viewport transformation, the near plane is 0.0 and the far plane is 1.0.