Transform mouse to screen positions

Question

I am using Open GL to make a game with a trimetric projection. I have the following as the view-projection transform:

float aspect = 1024.f/768.f;
glm::ortho<float>(-aspect*5, aspect*5, -5, 5, 0, 20) * glm::lookAt(glm::vec3(-std::sin(M_PI*36/180.f)*10,sin(M_PI*34/180.f)*10,10), glm::vec3(), glm::vec3(0,1,0));

The screen resolution is 1024x768. I want to map the mouse to positions on the screen. For example, tiles in my game are 1x1. When I hover over the first tile (which originates in the center of the screen) I want the position of the mouse to be between (0,0) and (1,1). I am not sure how to do this. What I got so far was to transform to a linear view of the screen (i.e. the orthogonal projection) by doing this:

glm::vec4 mousePos(x/768*10-apsect*5, 0, y/768*10-5, 0);

However I have no clue of where to go from there.

Answer 1

This is something very similar to an isometric interface for an iPad project I'm working on. That is the same screen resolution as your project. I was building other project code a couple days ago, when I first read your question. But the interface code for picking objects needed to be built. So it made sense to try developing the code. I setup a test using a blank coloured checkerboard to fit your question.

Here is a quick video demonstrating it. Please pardon the ugly look of it. Also please note about the video, the values following the cursor are are integers, but the code produces float values. There was another function I created to do that extra work for my own use. If that's something you'd like, I'll include it in my answer.

http://youtu.be/JyddfSf57ic

This code is deliberately verbose, because I am rusty at the math. So I have spent many hours the last couple days re-learning the dot and cross products, also reading barycentric code but deciding against it.

Last thing before the code: there is an issue with your projection matrix in the question. You are including the camera transform in the projection matrix. Technically that is allowed, but there are many resources online that suggest it's bad practice.

Hope this helps!

void calculateTouchPointOnGrid(CGPoint screenTouch) {
    float backingWidth = 1024;
    float backingHeight = 768;
    float aspect = backingWidth / backingHeight;
    float zNear = 0;
    float zFar = 20;
    glm::detail::tvec3<float> unprojectedNearZ;
    glm::detail::tvec3<float> unprojectedFarZ;

    /*
     Window coordinates, including zNear 
     This code uses zNear and zFar as two arbitrary values of 
             magnitude along the direction (vector) of the camera's 
             view (affected by projection and model-view matrices) that enable
     determining the line/ray, originating from screenTouch (or 
             mouse click, etc) that later intersects the plane.
     */

    glm::vec3 win = glm::vec3( screenTouch.x, backingHeight - screenTouch.y, zNear);


    // Model & View matrix

    glm::detail::tmat4x4<float> modelTransformMatrix = glm::detail::tmat4x4<float>(1);
    modelTransformMatrix = glm::translate(modelTransformMatrix, glm::detail::tvec3<float>(0,0,-1));
    modelTransformMatrix = glm::rotate(modelTransformMatrix, 0.f, glm::detail::tvec3<float>(0,1,0));
    modelTransformMatrix = glm::scale(modelTransformMatrix, glm::detail::tvec3<float>(1,1,1));

    glm::detail::tmat4x4<float> modelViewMatrix = lookAtMat * modelTransformMatrix;


    /*
     Projection
    */
    const glm::detail::tmat4x4<float> projectionMatrix =
        glm::ortho<float>(-aspect*5, aspect*5, -5, 5, 0, 20);


    /* 
     Viewport
    */      
    const glm::vec4 viewport = glm::vec4(0, 0, backingWidth, backingHeight);

    /*
     Calculate two points on a line/ray based on the window coordinate (including arbitrary Z value), plus modelViewMatrix, projection matrix and viewport
    */
    unprojectedNearZ = glm::unProject(win,
                                 modelViewMatrix,
                                 projectionMatrix,
                                 viewport);
    win[2] = zFar;
    unprojectedFarZ = glm::unProject(win,
                                 modelViewMatrix,
                                 projectionMatrix,
                                 viewport);


            /*
     Define the start of the ray
     */

    glm::vec3 rayStart( unprojectedNearZ[0], unprojectedNearZ[1], unprojectedNearZ[2] );

    /*
     Determine the vector traveling parallel to the camera from the two
      unprojected points
    */

    float lookatVectX = unprojectedFarZ[0] - unprojectedNearZ[0];
    float lookatVectY = unprojectedFarZ[1] - unprojectedNearZ[1];
    float lookatVectZ = unprojectedFarZ[2] - unprojectedNearZ[2];

    glm::vec3 dir( lookatVectX, lookatVectY, lookatVectZ );



    /*
     Define three points on the plane that will define a triangle.
     Winding order does not matter.
     */

    glm::vec3 p0(0,0,0);
    glm::vec3 p1(1,0,0);
    glm::vec3 p2(0,0,1);

    /*
     And finally the destination of the calculations
     */

    glm::vec3 linePlaneIntersect;


    if (cartesianLineIntersectPlane(rayStart, dir, p0, p1, p2, linePlaneIntersect)) {
        // do work here using the linePlaneIntersect values
    }
}


bool cartesianLineIntersectPlane(glm::vec3 rayStart, glm::vec3 dir,
                                 glm::vec3 p0, glm::vec3 p1, glm::vec3 p2, glm::vec3 &linePlaneIntersect) {

    /*
     Create edge vectors to form the plane normal
     */
    glm::vec3 edge1 = p1 - p0;
    glm::vec3 edge2 = p2 - p0;

    /*
     Check if the ray direction is parallel to plane, before continuing
     */

    glm::vec3 perpendicularvector = glm::cross(dir, edge2);


    /*
     dot product of edge1 on perpendicular vector
     if orthogonal (approximately 0) then ray is parallel to plane, has 0 or infinite contact points
     */

    float det = glm::dot(edge1, perpendicularvector);
    float Epsilon = std::numeric_limits<float>::epsilon();
    if (det > -Epsilon && det < Epsilon)
        // Parallel, return false
        return false;


    /*
     Calculate the normalized/unit normal vector
     */

    glm::vec3 planeNormal = glm::normalize( glm::cross(edge1, edge2) );


    /*
     Calculate d, the dot product of the normal and any point on the plane.
     D is the magnitude of the plane's normal, to the origin.
     */

    float d = planeNormal[0] * p0[0] + planeNormal[1] * p0[1] + planeNormal[2] * p0[2];


    /*
     Take the x,y,z equations, ie:
     finalP.xyz = raystart.xyz + dir.xyz * t

     substitute them into the scalar equation for plane, ie:
     ax + by + cz = d

     and solve for t.  (This gets a bit wordy) t is the 
    multiplier on the direction vector, originating at raystart, 
    where it intersects the plane. 


     eg:

     ax + by + cz = d
     a(raystart.x + dir.x * t) + b(raystart.y + dir.y * t) + c(raystart.z + dir.z * t) = d
     a(raystart.x) + a(dir.x*t) + b(raystart.y) + b(dir.y*t) + c(raystart.z) + c(dir.z*t) = d
     a(raystart.x) + a(dir.x*t) + b(raystart.y) + b(dir.y*t) + c(raystart.z) + c(dir.z*t) - d = 0
     a(raystart.x) + b(raystart.y) + c(raystart.z) - d = - a(dir.x*t) - b(dir.y*t) - c(dir.z*t)
     (a(raystart.x) + b(raystart.y) + c(raystart.z) - d) / ( a(-dir.x) + b(-dir.y) + c(-dir.z) = t

     */

    float leftsideScalars = (planeNormal[0] * rayStart[0] +
                             planeNormal[1] * rayStart[1] +
                             planeNormal[2] * rayStart[2] - d);
    float directionDotProduct = (planeNormal[0] * (-dir[0]) +
                                 planeNormal[1] * (-dir[1]) +
                                 planeNormal[2] * (-dir[2]) );


    /*
     Final calculation of t, hurrah!
     */

    float t = leftsideScalars / directionDotProduct;


    /*
     This is the particular value of t for that line that lies
     in the plane. Then you can solve for x, y, and z by going
     back up to the line equations and substituting t back in.
     */

    linePlaneIntersect = glm::vec3(rayStart[0] + dir[0] * t,
                                   rayStart[1] + dir[1] * t,
                                   rayStart[2] + dir[2] * t
                                   );
    return true;
}

Answer 2

You have screen space coordinates and you want to convert it to model space.

You have to apply the inverse to go back to model space from screen space.

GLM has a nice function called unProject that does this for you see Example code 1.

The problem you have is that you see your mouse coordinate in screen space as a point. The position of the mouse should be seen as a ray which is looking head on into your cursor. If it's just a point you actually have very little information about it.

So if you have a 2d coordinate from your mouse as screen space you need to convert that 2d point to two different 3d coordinates with different z values).

You then unproject those 3d points to get two model space points. These two points is then your ray that represents your mouse. See example code 2.

Then from the 3d ray you can get back to the tiling coordinate, by simply doing an intersection of the ray with the tile plane. See example code 3.

Example code 1

I added the full code in this example so that you can play with values to see what happens.

#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>
#include <iostream>
#include <cmath>

using namespace glm;
using namespace std;

int main()
{
  //Our Matrices
  float aspect = 1024.f/ 768;
  vec4 viewPort = vec4(0.f,0.f,1024.f,768.f);
  mat4 projectionMatrix =  ortho<float>(0, 1024, 0, 768, 0, 20);
  mat4 modelWorldMatrix = lookAt(vec3(-sin(M_PI*36/180.f)*10,sin(M_PI*34/180.f)*10,10), vec3(), vec3(0,1,0)); 

  //Our position
  vec3 pos = vec3(1.0f,2.0f,3.0f);

  //Tranform it into screen space
  vec3 transformed = project(pos, modelWorldMatrix, projectionMatrix, viewPort);
  cout << transformed.x << " " << transformed.y << " " << transformed.z << endl;

  //Transform it back
  vec3 untransformed = unProject(transformed, modelWorldMatrix, projectionMatrix, viewPort);
  cout << untransformed.x << " " << untransformed.y << " " << untransformed.z << endl;

  //You'll see how the glm's unproject works

  return 0;
}

Example code 2

  //You have your screen space coordinates as x and y
  vec3 screenPoint1 = vec3(x, y, 0.f);
  vec3 screenPoint2 = vec3(x, y, 20.f);

  //Unproject both these points
  vec3 modelPoint1 = unProject(screenPoint1, modelWorldMatrix, projectionMatrix, viewPort);
  vec3 modelPoint2 = unProject(screenPoint2, modelWorldMatrix, projectionMatrix, viewPort);

  //This is your ray with the two points modelPoint1, modelPoint2

Example code 3

  //normalOfPlane is the normal of the plane. If it's a xy plane then the normal is vec3(0,0,1)
  //P0 is a point on the plane

  //L is the direction of your ray
  //L0 is a point on the ray
  vec3 L = modelPoint1 - modelPoint2;
  vec3 L0 = modelPoint1;

  //Solve for d where dot((d * L + L0 - P0), n) = 0
  float d = dot(P0 - L0,normalOfPlane) / dot(L, normalOfPlane);

  //Use d to get back to point on plane
  vec3 pointOnPlane = d * L + L0; 

  //Sound of trumpets in the background
  cout << pointOnPlane.x << " " << pointOnPlane.y << endl;