CODEWITHSUNDEEP
c++windowsheap-memoryhook
xezon
xezon

Reputation: 91

How to get or store heap size in RtlAllocateHeap for later use in RtlFreeHeap?

The core problem: An old Windows XP game freezes on newer Windows versions unless it is run with Windows XP compatibility mode.

Even through it is not essential to provide a code fix for this, I still like to make a fix for this freeze, because noobie users run in this issue very often and ask every time how they can fix it. The application executable has no source code of course, so debugging the freeze is not straight forward. However, since Windows XP compat mode "fixes" it, I wanted to find out how.

So I looked it up on the web and found there is this thing called Shim Engine that provides those various fixes through AppHelp.dll. I installed the Windows Compat Toolkit and tracked down that a Shim called FaultTolerantHeap removes the application freeze. So far so good.

There are not much information about this FTH, but this video with Mr Calinoiu from Microsoft give a good enough inside to what they did and to there 4 core strategies in FTH:

As stated in the video this is done on functions like RtlAllocateHeap and RtlFreeHeap from ntdll. So I though I give it a try.

Next step I hooked both of these functions with Microsoft Detours and got access without problem. I redirected the calls from my hooked function directly to the original functions in ntdll. Game runs fine, hooks are good.

Then, I tried Zeroing memory: Freeze still happens. Appending 8 bytes to every heap allocation: Freeze still happens. Disabling the Freeing of memory: Freeze is gone! Aha!

So the delayed freeing of Microsoft's FTH seems to be the "cure" for the freeze. So the next step is to implement that delayed free. What we need is a queue with a maximum size of bytes it accepts to take, before the next free will purge and free the first entries in the queue. Microsoft uses a 4MB buffer for its queue. So I could do just the same I thought. But I have a big problem: I need to keep track of the size of the heap allocations that will be free'ed, otherwise there is no way to tell how large my queue is. Plus I do not want to queue super huge heaps, for example the buffer for a bitmap image is very large.

For Microsoft's FTH he states that they append their magic bytes at the very end of each heap allocation. Like 30 bytes + the 8 bytes padding. Quite a lot. I thought I could simplify it by just adding some 4 byte value as kind of "just good enough" pattern to identify a heap from my RtlAllocateHeap hook (because I hook on run time) and another 4 bytes to store the size of each allocation, so that I can access it when the allocation reaches RtlFreeHeap. The current code:

#define FTH_MAGIC_BYTES 0xFEDCBA98ul
#define FTH_ALLOC_PADDING 0

...

static PVOID __stdcall RtlAllocateHeap_hook(PVOID HeapHandle, ULONG Flags, SIZE_T Size)
{
    if(Size==0)
    {
        return RtlAllocateHeap_orig(HeapHandle, Flags, Size);
    }
    // For our allocations we assume 32bit application:
    // treat everything with size of 4 bytes
    Size += 8+FTH_ALLOC_PADDING;
    PVOID HeapBase = RtlAllocateHeap_orig(HeapHandle, Flags, Size);

    if(HeapBase)
    {
        *(uint32*)((uint32)HeapBase+4) = FTH_MAGIC_BYTES;
        *(uint32*)((uint32)HeapBase) = Size;
        return (PVOID)((uint32)HeapBase+8);
    }
    return HeapBase;
};


static BOOL __stdcall RtlFreeHeap_hook(PVOID HeapHandle, ULONG Flags, PVOID HeapBase)
{
    if(HeapBase==0)
    {
        return RtlFreeHeap_orig(HeapHandle, Flags, HeapBase);
    }
    if(*(uint32*)((uint32)HeapBase-4)==FTH_MAGIC_BYTES)
    {
        HeapBase = (PVOID)((uint32)HeapBase-8);
    }
    return RtlFreeHeap_orig(HeapHandle, Flags, HeapBase);
};

It crashes the application on boot. Interestingly, when i run Debug it does not crash though. On Release it does. I stepped through the Assembly in Visual Studio and could not see a problem. I also monitored the freshly allocated to see if the bytes are written in the heap - and they are, at the locations I expect them. Yet it throws off the application, and I don't understand why. When I do not change the offset of HeapBase on return of RtlAllocateHeap_hook, the code will run just fine.

    if(HeapBase)
    {
        //*(uint32*)((uint32)HeapBase+4) = FTH_MAGIC_BYTES;
        //*(uint32*)((uint32)HeapBase) = Size;
        return HeapBase;
    }

As soon as I return with the offset, the application goes crashy again. I can store my bytes at the end and then return HeapBase without offset, but then it will be madness to find that alignment when we hit RtlFreeHeap_hook. I wonder how Microsoft did that, unless they know the size of the heap of course. The only solutions I see is to get the above method right, find out how to retrieve the size of the heap the Microsoft way, or try my luck on hooking malloc or new.

Any ideas? Maybe I am missing something very simple :-)

Thanks.

Callstack sample:

ntdll.dll!76f030bd()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] msvcrt.dll!7498f480()
msvcrt.dll!7498f4e6()
msvcrt.dll!7498f52c()
comctl32.dll!73656c62()
game.dat!00413b90()
game.dat!00413fc8()
msvcrt.dll!7499edc8()
msvcrt.dll!7498a53a()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a6a9()
msvcrt.dll!7498d255()
msvcrt.dll!7498d272()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a53a()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498a5d6()
msvcrt.dll!7498dfd5()
game.dat!0041d4ed()
game.dat!006f0048()
game.dat!004af57d()
game.dat!00413b90()
game.dat!00413fc8()
game.dat!00413b90()
game.dat!00413fc8()
game.dat!004140d8()
game.dat!008fa9ad()
game.dat!0041f018()

EAX = 0262CE78 EBX = 00000000 ECX = 7C7EC8D9 EDX = 00000088 ESI = 00B70000 EDI = 00000000 EIP = 76F030BD ESP = 001886F8 EBP = 00188704 EFL = 00210202

7C7EC8E9 = ????

Upvotes: 0

Views: 2436

Answers (1)

xezon
xezon

Reputation: 91

After more testing and reading I saw in MSDN that kernel32.dll also exports Heap functions. Kernel32 has a function HeapSize that returns what we need. It seems that the Kernel functions just channel through the ntdll counterparts.

Therefore I suspected that ntdll should have a counterpart to HeapSize. I looked at the exports of ntdll and found RtlSizeHeap. This function is undocumented on MSDN.

A search on google returned this page from ntinternals.net. So you can either use the kernel function HeapSize or the ntdll function RtlSizeHeap by getting its pointer with GetProcAddress.

Upvotes: 1

Related Questions