_CrtMemDumpAllObjectsSince not returning expected results

Question

I'm using _CrtMemCheckpoint and _CrtMemDumpAllObjectsSince to track possible memory leaks in my dll.

In DllMain when DLL_PROCESS_ATTACH is detected an init function is called which calls _CrtMemCheckpoint(&startState) on the global _CrtMemState variable startState. When DLL_PROCESS_DETACH is detected an exit function is called that calls _CrtMemDumpAllObjectsSince(&startState). This returns

ExitInstance()Dumping objects ->
{8706} normal block at 0x07088200, 8 bytes long.
 Data: <p v     > 70 FF 76 07 01 01 CD CD 
{8705} normal block at 0x07084D28, 40 bytes long.
 Data: <                > 00 00 00 10 FF FF FF FF FF FF FF FF 00 00 00 00 
{4577} normal block at 0x070845F0, 40 bytes long.
 Data: <dbV             > 64 62 56 0F 01 00 00 00 FF FF FF FF FF FF FF FF 
{166} normal block at 0x028DD4B8, 40 bytes long.
 Data: <dbV             > 64 62 56 0F 01 00 00 00 FF FF FF FF FF FF FF FF 
{87} normal block at 0x02889BA8, 12 bytes long.
 Data: < P          > DC 50 90 02 00 00 00 00 01 00 00 00 

So far so good, except the last three entries (4577, 166 and 87) are also in startState. I.E. If I run _CrtDumpMemoryLeaks() in my Init function and in my Exit function those entries are in both lists.

The documentation says this:

_CrtMemDumpAllObjectsSince uses the value of the state parameter to determine where to initiate the dump operation. To begin dumping from a specified heap state, the state parameter must be a pointer to a _CrtMemState structure that has been filled in by _CrtMemCheckpoint before _CrtMemDumpAllObjectsSince was called.

Which makes me believe that items tracked in startState would be excluded from the output. At the end of the Init function where _CrtMemCheckpoint is called there have been about 4700 allocation calls. Shouldn't _CrtMemDumpAllObjectsSince only dump objects allocated after that checkpoint call?

What have I missed?

Answer 1

Short
This is apparently strange only, but it does the job (in part), however in an overzealous mode.
These functions are decades old, so not buggy but not completely well designed.

Truth is there is something in the "old" state that change after your "since" state.
So the question is "yes it does reflect a change since, but is it a lethal leak?"
This frequent and amplified with delayed init for DLL.
Also by a lot of complex objects like map/string/array/list which does delay allocation of internal buffer.
Bad news being that nearly all complex object declared as "static" are in fact inited on first use.

So theses change ought to be shown in _CrtMemDumpAllObjectsSince because they changed their memory alloc.
Unfortunately the display is so crude and unfiltered that it also show too many irrelevant blocks (not modified).

Typical biggest culprit is use of "realloc" that change state of old alloc

This may even look stranger as they may disappear,
For example when a genuine malloc is made after stat snapshoot, because this one will do a kind of 'reset' of the low water marker used for dump, setting it to a higher level. And this magically make a bunch of your "extra display" to disappear.
Behavior is even more erratic if you are doing Multithreading, as it becomes easily non repetitive.

Note:
The fact that it doesn't show a file name and line number is a sign that it's dealing here with a preinit.
So culprits are most likely static complex objetc inited BEFORE main() (or InitInstance();)

Long:
_CrtMemDumpAllObjectsSince is painful!
And in fact, can be so clutter of non-useful info that it defies the purpose for a day/day simple use of _CrtMemDumpAllObjectsSince. (In the good essence it inspires)
Workaround
None simple!
You may try to do a malloc then free AFTER you do your "since" state snapshoot, in order to tease this marker.
But to be safer and more in control, unfortunetaly I did saw way around writting my own _MyCrtMemDumpAllObjectsSince that dump from the original MS structure. This was inspired by static void __cdecl dump_all_object_since_nolock(_CrtMemState const* const state) throw()
( see "debug_heap.cpp") Copyright Microsoft!

Code available "as is" more for inspiration.

But before some explanation on the way _CrtMemState works:
A _CrtMemState state does have a pointer 'pBlockHeader' which is usually a link of a double linked list of '_CrtMemBlockHeader*'
This list is in fact more than a snapshoot at a time it is build, but a selection (unclear how) of all the memory blocks in use, arranged in such a way that the "current state" is directly pointed to by 'pBlockHeader'
So that going
-> _block_header_prev allows the exploration of older blocks
-> _block_header_next allows the exploration of newer blocks (the Juice you look for in the concept "Since" but very dangerous as there is no end marker)

The TRICKY part:
MS maintain a vital internal static _CrtMemBlockHeader* called __acrt_first_block This __acrt_first_block continuously changes during alloc and realloc

However, _MyCrtMemDumpAllObjectsSince dump does start from this __acrt_first_block and go forward (use _block_header_next) until finding a NULL ptr
The first block handled is decided by this __acrt_first_block, and the 'state' you sent is not more than a STOP of the dump.

Otherwise said _CrtMemDumpAllObjectsSince doesn't really dump the "since" state
But dump from __acrt_first_block as it is to your "since" state.

The 'for' loop is overkilling by showing blocks from a 'start'(since) to an 'end'(oldest modified since).
This makes sense but this does also encompass dumping blocks that have NOT been modified. Showing things we don't care about.

MS structure is clever and can be used directly, while it is not guaranteed that Microsoft will maintain the same vital structure _CrtMemBlockHeader in future
But over the last 15 years I haven't seen a bit of change in it (nor do I foresee any reason they would change a strategical and critical structure.)

I dislike copy/paste of MS code and resolve linker with my piggyback code
So the workaround I used is based on the capability to intercept text message sent to "Output" windows, decoding and storing ALL information in my own bank

Structurally below gives an idea of the intercepts using a static struct under lock to store all infos

_CrtSetReportHook2(_CRT_RPTHOOK_INSTALL,MyReportHookDumpFilter);
_CrtMemDumpAllObjectsSince(state);  // CONTRARY to what it says, this thing seems to dump everything until old_state
_CrtSetReportHook2(_CRT_RPTHOOK_REMOVE,MyReportHookDumpFilter);
_MyReportHookDumpFilterCommand(_CUST_SORT,NULL);
_MyReportHookDumpFilterCommand(_CUST_DUMP,NULL);

The `_MyReportHookDumpFilterCommand` does check the preexistence of blocks that are NOT modified at all and avoids displaying those during it's Dump phase
Take it as inspiration of code to ease display.

If anybody have simpler way to use it, please share!