MOUSEEVENTF_MOVE doesn't move to correct coordinates in-game

Question

Feel like I am bashing my head against the wall here so wanted to reach out and see if there is some easy solution I am missing first before I go crazy.

The problem:

• I was tasked to write a TAS for an older directx11 game for a charity event. I want to detect a pixel color, and move to that pixel. I have pixel detection working via OpenCV but the actual movements to that pixel do not line up.
• I found a function that does what I want, but it uses a fixed number to try and modify the movement by that I can't figure out for my game.
• My mouse dpi does not affect the movement, but the in-game sensitivity does.
• The game does not like SetCursorPos and seems to reject that input.

Code:

#include <iostream>
#include <Windows.h>

using namespace std;

//Magic number defines
const float constant = 0.116f;
float mouseSensitivity = 10.0f;
float modifier = mouseSensitivity * constant;

int centerScreenX = GetSystemMetrics(SM_CXSCREEN) / 2;
int centerScreenY = GetSystemMetrics(SM_CYSCREEN) / 2;

//Move mouse to center of screen
void centerMouse() {
    SetCursorPos(centerScreenX, centerScreenY);
    cout << "Moving to center of screen at(" << centerScreenX << ", " << centerScreenY << ")" << endl;
}

//Calculates actual coordinates for mouse movement based on sensitivity and a constant.
void calibrateCoordinates(int& x, int& y)
{
    if (abs(x) < 5)
        x = 0;
    else {
        x = x - centerScreenX;
        x = (int)((float)x / modifier);
    }

    if (abs(y) < 5)
        y = 0;
    else
    {
        y = y - centerScreenY;
        y = (int)((float)y / modifier);
    }
    cout << "Coordinates needed to move by (" << x << ", " << y << ")" << endl;
}


// Moves to x,y coordinates after processed into actual coordinates based on sensitivity and a constant.
void moveTo(int x, int y)
{
    SetProcessDPIAware();
    calibrateCoordinates(x, y);
    mouse_event(MOUSEEVENTF_MOVE, x, y, 0, 0);

    //Sanity check where the mouse ended up at
    POINT p;
    if (GetCursorPos(&p))
    {
        cout << "Mouse ended up at (" << p.x << ", " << p.y << ")" << endl;
    }
}


int main() {
    while (true) {
        // Check if the F19 button is pressed
        if (GetAsyncKeyState(VK_F19) & 0x8000) {

            //Hardcoded values of pixel we need to move to. Handled automatically via OpenCV in the real code. Simplified here
            int xTo = 784;
            int yTo = 686;

            //Centers mouse to line up cursor with crosshair
            centerMouse();

            //Tries to move to coords
            moveTo(xTo, yTo);
        }
    }
    return 0;
}

Output:

Matched pixel found at (784, 686)[4, 20, 222, 255] Moving to center of screen at (1280, 720) Coordinates needed to move by (-271, -20) Mouse ended up at (1009, 700)

The mouse should have ended up at (1012, 649) for the cross-hair to line up with the pixel I want. Do I just need to keep experimenting to find the magic number that it works with? Or is there any easier way to do this? Thanks

Answer 1

First of all, you appear to be passing absolute coordinates to the function mouse_event, but that function is interpreting them as coordinates relative to the last mouse position, because you are not passing the MOUSEEVENTF_ABSOLUTE flag to mouse_event. Therefore, if you want to pass absolute coordinates, you should also pass that flag. Also, Microsoft recommends that you use SendInput instead of mouse_event.

Another issue is the following:

The official Microsoft documentation for the function GetSystemMetrics states the following:

This API is not DPI aware, and should not be used if the calling thread is per-monitor DPI aware. For the DPI-aware version of this API, see GetSystemMetricsForDPI. For more information on DPI awareness, see the Windows High DPI documentation.

Therefore, the results of the lines

int centerScreenX = GetSystemMetrics(SM_CXSCREEN) / 2;
int centerScreenY = GetSystemMetrics(SM_CYSCREEN) / 2;

are possibly incorrect. I suggest that you change these lines to the following:

int centerScreenX = GetSystemMetricsForDpi( SM_CXSCREEN, GetDpiForSystem() ) / 2;
int centerScreenY = GetSystemMetricsForDpi( SM_CYSCREEN, GetDpiForSystem() ) / 2;

However, the function GetDpiForSystem will simply return the value 96 unless the process or thread is made DPI-aware beforehand. Therefore, it is important that you call the function SetProcessDPIAware before calling the function GetDPIForSystem. Alternatively, you can set the default DPI awareness through an application manifest setting, as described here.

According to this Microsoft documentation, there are currently 4 different modes of DPI awareness:

1. Unaware
2. System
3. Per-Monitor
4. Per-Monitor V2

Since you are using the function SetProcessDPIAware, you are setting the DPI awareness mode of your process to "System". This mode will only make your process aware of the DPI on the primary monitor at the time the user logged in. If the DPI was changed since then, then your process will be unaware of these changes. For this reason, you may want to consider using "Per-Monitor V2" DPI-awareness instead, by using the function SetProcessDpiAwarenessContext or SetThreadDpiAwarenessContext. However, this will require your application to handle WM_DPICHANGED messages.

Answer 2

Disclaimer: Sorry if this isn't an answer but I can't comment yet because of my low rep.

1. If it's a game, have you tried taking the DX9/11's version of "GetSystemMetrics"? I had the same problem on DX9 aimbot years ago where the mouse aims slightly to the right. Now I can't give the exact answer since I've never done DX11 hooking but this is the mouse_move code for my aimbot:

mouse_event(MOUSEEVENTF_MOVE, coordinates.x - interfaceinfo->D3D9Viewport.Width * 0.5f, coordinates.y - interfaceinfo->D3D9Viewport.Height * 0.5f, 0, 0);

The coordinates is the target's position, while interfaceinfo->D3D9Viewport.Width/Height is the screen info given by the DX9, in this case using **IDirect3DDevice9::GetViewport**

HRESULT WINAPI Renderer::hkPresent(LPDIRECT3DDEVICE9 pDevice, CONST RECT* pSourceRect, CONST RECT* pDestRect, HWND hDestWindowOverride, CONST RGNDATA* pDirtyRegion) {
    renderer->Device_D3D9 = pDevice;
    Device_D3D9->GetViewport(&interfaceinfo->D3D9Viewport);
    return renderer->oPresent(pDevice, pSourceRect, pDestRect, hDestWindowOverride, pDirtyRegion);
}

2. If it's a game, have you tried moving the mouse by hooking DINPUT functions. By hooking to GetDeviceState and GetDeviceData, you can emulate keyclicks and mouse clicks/movement without moving your actual mouse (unlike sendinput and mouse_event)

HRESULT WINAPI Renderer::hkGetDeviceState(IDirectInputDevice8A* DIDevice, DWORD cbData, LPVOID *lpvData) {
    HRESULT hResult = renderer->oGetDeviceState(DIDevice, cbData, lpvData);
    
    if (interfaceinfo->bCurrentWindow >= 1) {
        if (hResult == DI_OK) {
            for (auto i = 0; i < cbData; i++) {
                lpvData[i] = 0;
            }
        }
    }

    static BYTE buffer[256];
    if (true) {
        buffer[DIK_W] = LOBYTE(0x80);
        buffer['w'] = LOBYTE(0x80);
        buffer['W'] = LOBYTE(0x80);
        cbData = 256;
        memcpy(lpvData, buffer, cbData);
    }
    else {
        hResult = renderer->oGetDeviceState(DIDevice, cbData, lpvData);
    }
    
    return hResult;
}

HRESULT WINAPI Renderer::hkGetDeviceData(IDirectInputDevice8A* DIDevice, DWORD cbObjectData, LPDIDEVICEOBJECTDATA rgdod, LPDWORD pdwInOut, DWORD dwFlags) {
    HRESULT ret = renderer->oGetDeviceData(DIDevice, cbObjectData, rgdod, pdwInOut, dwFlags);
    if (ret == DI_OK) {
        for (DWORD i = 0; i < *pdwInOut; ++i) {
            if (interfaceinfo->bCurrentWindow >= 1) {
                *pdwInOut = 0;
            }
            else {
                switch (rgdod[i].dwOfs) {
                case DIK_W:
                        rgdod[i].dwData = LOBYTE(0x80);
                    break;
                case DIK_SPACE: {
                    static byte last_written = 0;
                    rgdod[i].dwData = HIBYTE(last_written ? 0x01 : 0x00);
                    last_written = last_written ? 0x00 : 0x01;
                    break;
                }
                }
            }
        }
    }
    return ret;
}

Answer 3

GetSystemMetrics function is not DPI aware. Use GetSystemMetricsForDpi instead to get a correct result.