Getting function addresses from a DllMain hook

Question

Creating DirectX objects directly from DllMain() fails. I have tried starting a new thread, create the objects there and them get vtable/addresses of the functions I'm about to hook.

I'm using Microsoft Detours, DetourTransactionCommit() cannot be called from another thread than DllMain thread.

I have also tried waiting for the thread to finish collecting the addresses, like this:

BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
    switch (dwReason)
    {
        case DLL_PROCESS_ATTACH:
        {          
            std::thread(InitializeHooks).detach();            
            while (!_done)
            {
                std::this_thread::sleep_for(std::chrono::milliseconds(100));
            }
            // ...
            break;
        }
    }
    
    return TRUE;
}

But doing this, it gets stuck whenever it tries to call any DirectX function.

I'm getting the DirectX function addresses like this:

void InitializeHooks()
{
    if (HMODULE d3d9Module = GetModuleHandle("d3d9.dll"))
    {
        typedef IDirect3D9* (WINAPI* Direct3DCreate9Func)(UINT SDKVersion);
        Direct3DCreate9Func d3d9Create = (Direct3DCreate9Func)GetProcAddress(d3d9Module, "Direct3DCreate9");

        if (d3d9Create)
        {
            IDirect3D9* d3d9 = d3d9Create(D3D_SDK_VERSION);
            if (d3d9)
            {
                D3DPRESENT_PARAMETERS d3dpp = {};
                d3dpp.Windowed = TRUE;
                d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
                d3dpp.hDeviceWindow = GetDesktopWindow();

                IDirect3DDevice9* device = nullptr;
                if (SUCCEEDED(d3d9->CreateDevice(
                    D3DADAPTER_DEFAULT,
                    D3DDEVTYPE_HAL,
                    GetDesktopWindow(),
                    D3DCREATE_HARDWARE_VERTEXPROCESSING,
                    &d3dpp,
                    &device)))
                {
                    void* deviceVTable = *(void**)device;
                    HookVirtualMethod(
                        (PVOID*)&originalD3D9Present,
                        (PVOID)HookedD3D9Present,
                        deviceVTable,
                        17
                    );

                    device->Release();
                }
                d3d9->Release();
            }
        }
    }

    // ...
}

I have tested collecting the address from a different process and my hooks worked correctly:

void HookVirtualMethod(PVOID* ppOriginal, PVOID pDetour) 
{
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach(ppOriginal, pDetour);
    r = DetourTransactionCommit();
}
// ...

    originalD3D9Present      = reinterpret_cast<D3D9Present>(0x00007ffd891c8c90);
    originalD3D11Draw        = reinterpret_cast<D3D11Draw>(0x00007ffdbd7cd0b0);
    originalD3D11DrawIndexed = reinterpret_cast<D3D11DrawIndexed>(0x00007ffdbd7cdc30);
    originalDXGIPresent      = reinterpret_cast<DXGIPresent>(0x00007ffdbe2e18c0);
    
    
    HookVirtualMethod((PVOID*)&originalD3D9Present, (PVOID)HookedD3D9Present);
    HookVirtualMethod((PVOID*)&originalD3D11Draw, (PVOID)HookedD3D11Draw);
    HookVirtualMethod((PVOID*)&originalD3D11DrawIndexed, (PVOID)HookedD3D11DrawIndexed);
    HookVirtualMethod((PVOID*)&originalDXGIPresent, (PVOID)HookedDXGIPresent);

How I could get the addresses from inside the DLL in this case?

Answer 1

When a DLL is loaded, between the call to LoadLibrary(Ex) and its return, there are some steps which the Operating System does. One of those is to load the further dependencies of other Dlls, and to initialize the Dynamic data for the DLL which needs code to execute.

During this process the number of Loaded Dlls is sort of indeterminate - if my DLL tried to load a Dll, which was being loaded as an already calculated dependency then, it may load the Dll twice or not at all.

The Windows system has the LoaderLock this lock ensures only one thread is able to modify the loaded Dlls, and is owned at the point when LoadLibrary starts its work.

This means that other threads which Load libaries, will be unable to until the LoadLibrary returns, and that is the cause of your deadlock when you are using Detours.

GetProcAddress, GetModuleHandle and various other queries on Dll state will also be blocked on the LoaderLock

The other thread can't progres, because DllMain is active, and the LoaderLock held. DllMain thread can't progress because it is waiting for the thread to complete its work.

The C++ standard brushes very loosely over the behaviour of LoadLibrary and dlopen. At the end of the calls, the code is loaded, but during the calls, makes no guarantees of the environment the code runs in.

global data in a Dll is also initialized (if it needs a function) during this period, so a global variable will not help you.

A function static is created when the function is called, so it may be possible to use that as a deferral to when the Dll is loaded.