SetWindowsHookEx not calling my callback?

Question

Well I tried different solution to my problem but It just doesn't work.

I call SetWindowsHookExA and then when I press a key the messagebox is not shown. What to do?

this is my code (this is a DLL which is loaded by another DLL which is loaded by the program):

#include <Windows.h>

HINSTANCE gl_hThisInstance = NULL;
HHOOK hHook = NULL;

LRESULT CALLBACK KeyHit(int code,WPARAM wParam,LPARAM lParam);

BOOL APIENTRY DllMain( HANDLE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved)
{
    switch (ul_reason_for_call)
    {
        case DLL_PROCESS_ATTACH:
         gl_hThisInstance = (HINSTANCE)hModule;
         hHook = SetWindowsHookExA(
            WH_KEYBOARD,
            KeyHit,
            //(HWND)gl_hThisInstance//not working
            0,//not working
            //(DWORD)gl_hThisInstance//not working
            //GetCurrentThreadId()//even not working with this
            0//not working
            );
        break;
    }
    return TRUE;
}

LRESULT CALLBACK KeyHit(int code,WPARAM wParam,LPARAM lParam)
{
    MessageBox(0,"PRESSED","PRESSED",0);
    return CallNextHookEx(hHook,code,wParam,lParam);
}

Answer 1

I know this is super old but i just ran across this issue myself the other day, and as the other answer suggested, you cant call SetWindowsHookEx() from the DLLMain function or other threads.

BUT, thats because the DLL main function runs in its own thread, separate from the main thread. And thats relevant because apparently it seems that when you call SetWindowsHookEx(), as soon as the thread that you called that from, expires, then it automatically unhooks for some reason (and i couldn't find this documented anywhere on the internet).

So if you call SetWindowsHookEx() from any thread other than the main one of the target process (assuming you can do that) then theres always a possibility that your hook will be undone before the main process itself expires. And in this instance, once the DLLMain function returns, it terminates the thread and consequently removes the hook.

So to fix this i call the hook from a new thread, and then i suspend the thread indefinitely, which should mean the hook only gets removed when the process itself closes.

The fixed version of the code from the question could look something possibly like this

HHOOK hHook = NULL;
LRESULT CALLBACK KeyHit(int code, WPARAM wParam, LPARAM lParam){
    // replace with whatever logic you want to run in the target process when keyboard event is called
    cout << "keyboard event detected.\n";
    return CallNextHookEx(hHook, code, wParam, lParam);
}
void ThreadHookEvents() {
    DWORD thread_id = 0; // set this to the thread id of your main/hwnd thread
    HHOOK hHook = SetWindowsHookExA(WH_KEYBOARD, (HOOKPROC)KeyHit, 0, thread_id);
    // then pause the thread so the hook never expires
    SuspendThread(GetCurrentThread());
}
BOOL APIENTRY DllMain(HANDLE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved){
    switch (ul_reason_for_call){
    case DLL_PROCESS_ATTACH:
        CreateThread(nullptr, 0, (LPTHREAD_START_ROUTINE)ThreadHookEvents, 0, 0, 0);
        break;
    }
    return TRUE;
}

As a bonus incase someone actually reads this, i put together another small function to get the thread ID of a window (not specifically the main one, if multiple) from the process, which you could just plug right into the above script for getting the thread_id

DWORD GetAWindowThreadID() {
    auto EnumWindowsProc = [](HWND hwnd, LPARAM lParam) -> BOOL {
        DWORD windowProcessId;
        DWORD thread_id = GetWindowThreadProcessId(hwnd, &windowProcessId);
        if (windowProcessId == GetCurrentProcessId())
            *(DWORD*)lParam = thread_id;
        return TRUE; 
    };

    DWORD target_thread = 0;
    EnumWindows(EnumWindowsProc, (LPARAM)&target_thread);
    return target_thread;
}

Answer 2

I got issues with the hooking before. Not really an issue, but the way I did it was not supposed to. First of all, you should have 2 exported functions from the DLL, SetHook and RemoveHook. The SetHook function will call the SetWindowsHookEx() from there. If you ever try to call the SetWindowsHookEx() from within a thread or DLLMain of your DLL, the function returns no errors, but the callback function will never be called. It took me sometimes to figure it out.

Posted here is my working code to catch WH_GETMESSAGE, you can reference from here.

Here is my working exported SetHook() function from the DLL.

bool __declspec(dllexport) __stdcall SetHook(DWORD myWnd)
{
    mySavedHook = SetWindowsHookEx(WH_GETMESSAGE,
        GetMsgProc,
        infoContainer.DllHModule,
        myWnd);

    int errorID = GetLastError();
    if (errorID != 0)
    {
        MessageBoxA(NULL, "Failed to implement hook", "Failed", 0);
        MessageBoxA(NULL, to_string(errorID).c_str(), "Error ID", 0);
        return false;
    }
    else
    {
        return true;
    }
}

infoContainer.DllHModule: instance of the DLL, which is the first parameter of DllMain(). myWnd: is my Thread ID (not Process ID) - Get it from GetWindowThreadProcessId(window_handle, NULL). To implement global hook, use 0 as myWnd.

And here is my callback function.

LRESULT CALLBACK GetMsgProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    if (nCode >= 0)
    {
        LPMSG msg = (LPMSG)lParam;
        if (msg->message == WM_CALLFUNCTION)
        {
            MessageBoxA(NULL, "Receive WM_CALLFUNTION", "Good news", 0);
        }

    }
    //Doesn't matter, just call this function and return it.
    return CallNextHookEx(_hook, nCode, wParam, lParam);
}

The callback function must have the CALLBACK keyword to work.

From the external application, call the SetHook() from the DLL and use the Thread ID as its param.