Memory overwrites in C++ code showing up in consistent locations

Question

I have a very un-scientific observation about memory overwrites and was curious if anyone else has noticed something similar, knows why, and/or can tell me why I wasn't really seeing what I thought I was seeing.

What I noticed was that for some C++ programs, when I have a memory overwrite bug in that program, it would usually (if not always) show up in a specific section of code which was often unrelated to the section of code with the bug. This is not a blanket observation. Not all C++ programs behave this way. But when I have one, it is pretty consistent. (No comment on why my code has enough memory overwrites that I have the opportunity to notice consistent-anything :) )

I'm not asking why a memory overwrite in function1 can show up in function2; that is understood. My observation is that over the life of a given program, we have discovered memory overwrites in function1, function2, function3, function4, and function5. But in each case, we discovered the problem because the code would crash in function6. Always in function6 and only in function6. None of those functions are related and do not touch anything that function6 uses.

Over my lifetime, I've encountered two C programs and one C++ program that behaved this way. These were years apart in unrelated systems and hardware. I just found it weird and wondered if anyone else has seen this. Plus, I suspect that I may be seeing the same pattern in a C++/JNI/Java program that I'm working on now, but it is young enough that I've not had enough hits to be sure of a pattern.

Answer 1

Perhaps the real question here is about what escalates "silent" memory corruption into an actual/formal crash (as opposed to "just" more subtle problems such as unexpected data values, that the user might or might not notice or recognize). I don't think that question can be answered generally, as it depends a lot on the specifics of the compiler, the code, the memory layout of the in-memory data structures, etc.

It can be said that in most modern (non-embedded) systems there is an MMU that handles translating virtual (i.e. per-process) memory addresses into physical memory addresses, and that many user-space crashes are the result of the MMU generating an unrecoverable page fault when the user program tries to dereference a virtual address that has no defined physical equivalent. So perhaps in this case function6() was trying to dereference a pointer whose value had been corrupted in such a way that the MMU couldn't translate it to a physical address. Note that the compiler often places its own pointers on the stack (to remember where the program's control flow should return to when a function returns), so a bad pointer dereference can happen even in code that doesn't explicitly dereference any pointers.

Another common cause for a crash would be a deliberately induced crash invoked by code that notices that the data it is working with is "in a state that should never happen" and calls abort() or similar. This can happen in user code that has assert() calls in it, or in system-provided code such as the code that manages the process's heap. So it could be that function6() tried to allocate or free heap memory, and in so doing gave the heap manager the chance to detect an "impossible state" in one of its data structures and crash out. Keep in mind that the process's heap is really just one big data structure that is shared by all parts of the program that use the heap, so it's not terribly surprising that heap corruption caused by one part of the program might result in a crash later on by another (mostly unrelated) part of the program that also uses the heap.