Why does this detour function crash the program?

Question

I have a proxy dxgi.dll and I'm trying to detour the Present function in the original dxgi.dll in order to render things on screen. The .dll is successfully loaded and the detour is placed. However the detour crashes the program as soon as my new Present is called. Keep in mind the .dlls and programs are 64-bit.

Below is an image of how the function looks in memory before modification (Start highlighted):

Okay so I just found out I'm not allowed to post images directly on here unless I have 10 reputation, so use this link (replace DOT): https://imgur DOT com/a/Jf53dYc

I am not sure exactly where it crashes, I believe the program keeps running for a little while, but it definetly crashes in the middle/soon after the detour Present is called, I know this because I can write the pointer to the SwapChain parameter to a file from inside the Present detour before it crashes.

I found the original Present function address using IDA. You can see what IDA says about the function on the picture in the imgur gallery.

I've been looking at the memory and been trying to figure out what is wrong, when I follow the jumps using Cheat engine they lead to the correct places, nevertheless something in the detour is making the program crash. The overriden opcodes also seem to be replaced properly.

I've tried to change the calling convention and return type on my Present function, I read in a dxgi hooking guide that the return type was a HRESULT, I tried changing to this to no avail. As for the calling convention I've tried WINAPI.

I've also looked a little bit into if the stack or registers are being corrupted by my function detour. However I'm not very good with assembly and I can't say for sure if this is the case.

I have a class named Core that takes care of the hooking, here is the header file with some relevant definitions:

#pragma once
#include <iostream>
#include <Windows.h>
#include <intrin.h>
#include <dxgi.h>
#include <fstream>

// Seems my C++ doesn't have QWORD predefined, defining it myself
typedef unsigned __int64 QWORD; 

// Definition of the structure of the DXGI present function
typedef __int64 (__fastcall* PresentFunction)(IDXGISwapChain *pSwapChain, UINT SyncInterval, UINT Flags);

class Core
{
private:
    QWORD originalDllBaseAddress;
    QWORD originalPresentFunctionOffset;

public:
    void Init();
    bool Hook(PresentFunction originalFunction, void* newFunction, int bytes);
    ~Core();
};

Init starts the process by getting the relevant addresses:

void Core::Init()
{
originalDllBaseAddress = (QWORD)GetModuleHandleA("dxgi_.dll");
originalPresentFunctionOffset = 0x5070;
originalPresentFunction = (PresentFunction)(originalDllBaseAddress + (QWORD)originalPresentFunctionOffset);
Hook(originalPresentFunction, FixAndReturn, 14);
}

Hook tries to place a jump in the target address, I strongly believe the issue is somewhere in here, (comments have now changed my mind, it probably has something to do with assembly, registers or the stack) more specifically the assignments to originalFunction:

bool Core::Hook(PresentFunction originalFunction, void* newFunction, int length)
{
    DWORD oldProtection;

    VirtualProtect(originalFunction, length, PAGE_EXECUTE_READWRITE, &oldProtection);

    memset(originalFunction, 0x90, length);

    // Bytes are flipped (because of endianness), could alternatively use _byteswap_uint64()
    *(QWORD*)originalFunction = 0x0000000025FF;

    // The kind of jump I'm doing here seems to only use 6 bytes,
    // and then grabs the subsequent memory address,
    // I'm not quite sure if I'm doing this right
    *(QWORD*)((QWORD)originalFunction + 6) = (QWORD)newFunction;

    DWORD temp;
    VirtualProtect(originalFunction, length, oldProtection, &temp);

    originalPresentFunction = (PresentFunction)((QWORD)originalFunction + length);

    presentAddr = (QWORD)Present;
    jmpBackAddr = (QWORD)originalPresentFunction;

    return true;
}

I've tried many things when it comes to writing the bytes into memory, but none of them have fixed my problem.

The assignment to "originalPresentFunction" at the end of the function is the address that the detour will attempt to jump back to.

Here is the definition of the detour function, located in Core.cpp:

__int64 __fastcall Present(IDXGISwapChain *pSwapChain, UINT SyncInterval, UINT Flags)
{
    //The program crashes with and without these file writes.
    std::ofstream file;
    file.open("HELLO FROM PRESENT.txt");
    file << pSwapChain;
    file.close();

    return originalPresentFunction(pSwapChain, SyncInterval, Flags);
}

This is the function, when called, that causes a crash. As you can see, I am writing the pSwapChain parameter to a file here. I did this to test if the parameters are being passed from the original function. This write is successful, and the contents of the file look like a valid pointer. thus the crash happens after this write. FixAndReturn() is an assembly function.

includelib legacy_stdio_definitions.lib

.data
extern presentAddr : qword
extern jmpBackAddr : qword

; This performs instructions originally performed by dxgi.dll in the
; memory that we've replaced, and then returns

.code
    FixAndReturn PROC 
        call [presentAddr]
        mov [rsp+10h],rbx
        mov [rsp+20h],rsi
        push rbp
        push rdi
        push r14
        jmp qword ptr [jmpBackAddr]
    FixAndReturn ENDP
end

I have uploaded the entire code on Github if more code is needed: https://github.com/techiew/KenshiDXHook

Answer 1

It's been a while, I've been busy with other things but I've now made the detour function work successfully.

After looking at resources on the web and doing a lot of thinking. The answer is quite simple. In my FixAndReturn assembly code, all I need to do is jmp to the detour function, no call is needed. A call might unneccesarily change things we don't want to, and our detour function is already identical to the original function in terms of parameters and whatnot, so it will already read the parameters from the same place that the original function call placed them. This means a jmp will work just fine for running our detour function. No extra pushes or pops are needed in assembly for this to work.

Here is a basic overview of the process:

• Our hook is placed by placing a jmp to the beginning of our assembly code.
• Our assembly code immediately jumps to our detour/hooked function.
• When the detour function is finished, it returns a function call.

This function call uses a typedef which is identical to the original function we hooked. It looks like this:

typedef HRESULT (__fastcall* PresentFunction)(IDXGISwapChain *pSwapChain, UINT SyncInterval, UINT Flags);

Returning the function using the typedef is done like this, with the original argument values:

return ((PresentFunction)coreRef->newPresentReturn)(swapChain, syncInterval, flags);

Basically what's happening here is that the address following right after our second assembly code jmp instruction pointing to our detour function is being returned to and called as a function, thus we are jumping to the detour, jumping back, and executing the original code. (coreRef->newPresentReturn contains the address right after the jmp instruction).

We are now adhering to the calling convention of the original Present function, and the parameters we pass in are put in the right places, the registers and stack and whatever are not corrupted in any way.

Resource used: Guidedhacking.com - D3D11 barebones hook

Full code is on my Github: https://github.com/techiew/KenshiHook