Native access violation with .NET Application

Question

we have a problem with a .NET Application. It randomly on closing produces a native access violation (so not the .NET Exception).

• Windows Server 2012 x64 (Virtual Machine running on a blade system)
• .NET 4.0 Framework (installed 4.5)
• our application is .NET x86 application (not AnyCPU; not ASP.NET)
• Third Party native modules used SQL CE 3.5, and others

The error report from windows event log looks something like this (application and module names omitted and I translated the entries from German):

Exceptioncode: 0xc0000005
Offset: 0x00006a9e
Process ID: 0xfe8
...

I found out that 0xc0000005 is an access violation. Which could also occur due to a .NET NullReferenceException. We already created a full memory dump with ProcDump flag -ma when the windows error reporting dialog was open.

When I open this dump I cannot find any .NET modules loaded other than the .exe file itself. I am not an expert on native programming and only know some basics about WinDbg.

So what I did:

1. Open the dump with WinDbg

2. Setup the symbol path with

   .sympath SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols;C:\PathToMatchingPDBs\ForTheProgram
   

3. Reload all symbols with

   .reload /d /f
   
   Reloading current modules
   ...*** ERROR: Symbol file could not be found.  Defaulted to export symbols for sysfer.dll - 
   ...
   

4. ~* k outputs

   .  0  Id: 1560.19a4 Suspend: 0 Teb: ff20e000 Unfrozen
   ChildEBP RetAddr  
   0058f0e8 7744c752 ntdll!ZwWaitForMultipleObjects+0xc
   0058f26c 76d256c0 KERNELBASE!WaitForMultipleObjectsEx+0x10b
   0058f2e0 76d2586a kernel32!WerpReportFaultInternal+0x1c4
   0058f2f4 76cf7828 kernel32!WerpReportFault+0x6d
   0058f300 774d07c4 kernel32!BasepReportFault+0x19
   0058f39c 7762c11c KERNELBASE!UnhandledExceptionFilter+0x1f1
   0058f3a4 775f3334 ntdll!__RtlUserThreadStart+0x57
   0058f3b8 77691fd7 ntdll!_EH4_CallFilterFunc+0x12
   0058f3e0 77693612 ntdll!_except_handler4_common+0x8e
   0058f400 775f30f1 ntdll!_except_handler4+0x20
   0058f424 775f30c3 ntdll!ExecuteHandler2+0x26
   0058f4ec 775f2f2b ntdll!ExecuteHandler+0x24
   0058f4ec 00406a9e ntdll!KiUserExceptionDispatcher+0xf
   0058f844 7760ac69 MyProgram!COM+_Entry_Point <PERF> (MyProgram+0x6a9e)
   0058f888 7760ac3c ntdll!__RtlUserThreadStart+0x72
   0058f8a0 00000000 ntdll!_RtlUserThreadStart+0x1
   

5. The next part I am pretty unsure if it is correct what I did. I read somewhere that you can take the address of the column RetAddr and pass it to uf to see what the function at this point is doing. Is this correct? So I am expecting the following command to display the disassembly from the function at MyProgram+0x6a9e, see call stack above.

   >uf 0x00406a9e
   MyProgram!COM+_Entry_Point <PERF> (MyProgram +0x6a9e):
   00406a9e ff2500204000    jmp     dword ptr [MyProgram!COM+_Entry_Point <PERF> (MyProgram+0x2000) (00402000)]
   
   mscoree!_CorExeMain_Exported:
   707f4ddb 8bff            mov     edi,edi
   707f4ddd 56              push    esi
   707f4dde e80c2f0000      call    mscoree!ShellShim__CorExeMain (707f7cef)
   707f4de3 6a00            push    0
   707f4de5 8bf0            mov     esi,eax
   707f4de7 e84bc4ffff      call    mscoree!GetShimImpl (707f1237)
   707f4dec e93a800000      jmp     mscoree!_CorExeMain_Exported+0x11 (707fce2b)
   
   mscoree!_CorExeMain_Exported+0x11:
   707fce2b 83f801          cmp     eax,1
   707fce2e 750a            jne     mscoree!_CorExeMain_Exported+0x20 (707fce3a)
   
   mscoree!_CorExeMain_Exported+0x16:
   707fce30 6858378370      push    offset mscoree!g_wszShimImplDllPath (70833758)
   707fce35 e826350100      call    mscoree!DisplayMessageBoxForNoShimImpl (70810360)
   
   mscoree!_CorExeMain_Exported+0x20:
   707fce3a 8bc6            mov     eax,esi
   707fce3c 5e              pop     esi
   707fce3d c3              ret
   

   I am not sure how to interpret this. I looks like this is really the entry point of the application because of the CoreExeMain Exports? So not so much useful informations here, or? Only that the problem occurs on native code level; as far as I interpreted it.

And just for completeness:

.loadby sos clr
Unable to find module ‘clr’
!pe
No export pe found.

When I open the dump with Visual Studio there is also written no exception found.

So that’s how far I got. Maybe someone could help me to interpret this data and/or give me some hints what else I can try?

My questions:

• Can IL-Code generated with Reflection.Emit result in this kind of error?
• What can I do to further analyse the issue/find the cause?
• What does COM+_Entry_Point mean. Is this the main entry point of my application or does this have to do with some external component?

EDIT

@Jochen Kalmbach

.ecxr;kP

Minidump doesn't have an exception context
Unable to get exception context, HRESULT 0x80004002
ChildEBP RetAddr  
0058f0e8 7744c752 ntdll!ZwWaitForMultipleObjects+0xc
0058f26c 76d256c0 KERNELBASE!WaitForMultipleObjectsEx+0x10b
0058f2e0 76d2586a kernel32!WerpReportFaultInternal+0x1c4
0058f2f4 76cf7828 kernel32!WerpReportFault+0x6d
0058f300 774d07c4 kernel32!BasepReportFault+0x19
0058f39c 7762c11c KERNELBASE!UnhandledExceptionFilter+0x1f1
0058f3a4 775f3334 ntdll!__RtlUserThreadStart+0x57
0058f3b8 77691fd7 ntdll!_EH4_CallFilterFunc+0x12
0058f3e0 77693612 ntdll!_except_handler4_common+0x8e
0058f400 775f30f1 ntdll!_except_handler4+0x20
0058f424 775f30c3 ntdll!ExecuteHandler2+0x26
0058f4ec 775f2f2b ntdll!ExecuteHandler+0x24
0058f4ec 00406a9e ntdll!KiUserExceptionDispatcher+0xf
0058f844 7760ac69 MyProgram!COM+_Entry_Point <PERF> 
0058f888 7760ac3c ntdll!__RtlUserThreadStart+0x72
0058f8a0 00000000 ntdll!_RtlUserThreadStart+0x1b

Why minidump?

@Thomas W.

lm

start    end        module name
00400000 0040c000   MyProgram   (private pdb symbols)  c:\...\release\MyProgram.pdb
707f0000 7083a000   mscoree    (pdb symbols)          c:\symbols\mscoree.pdb\7608F9FF3C954E429A27D833164E4BEE2\mscoree.pdb
74b70000 74bdd000   sysfer     (export symbols)       sysfer.dll
76cc0000 76df0000   kernel32   (pdb symbols)          c:\symbols\wkernel32.pdb\CFACA818EC334A5EAD707CD86C847B4A1\wkernel32.pdb
77440000 774e6000   KERNELBASE   (pdb symbols)          c:\symbols\wkernelbase.pdb\CAE0056433064B78937D9022B20FA1102\wkernelbase.pdb
775b0000 77707000   ntdll      (private pdb symbols)  c:\symbols\wntdll.pdb\EC83D8DF555946E0B630133EBD9792662\wntdll.pdb

Edit 2

Additional information:

!heap
Index   Address  Name      Debugging options enabled
  1:   009c0000                
  2:   006b0000                
  3:   00770000                
  4:   00d00000


!heap -a
Index   Address  Name      Debugging options enabled
  1:   009c0000 
    Segment at 009c0000 to 00abf000 (0000b000 bytes committed)
  2:   006b0000 
    Segment at 006b0000 to 006bf000 (00003000 bytes committed)
    Segment at 00810000 to 0090f000 (00042000 bytes committed)
  3:   00770000 
    Segment at 00770000 to 0077f000 (00003000 bytes committed)
  4:   00d00000 
    Segment at 00d00000 to 00d3f000 (00001000 bytes committed)


!heap -l
Searching the memory for potential unreachable busy blocks.
Heap 009c0000
Heap 006b0000
Heap 00770000
Heap 00d00000
Scanning VM ...
Scanning references from 204 busy blocks (0 MBytes) ...
Entry     User      Heap      Segment       Size  PrevSize  Unused    Flags
-----------------------------------------------------------------------------
006b07c0  006b07c8  006b0000  006b0000        88       220         8  busy 
007707c0  007707c8  00770000  00770000        88       220         8  busy 
2 potential unreachable blocks were detected.

So what does that mean?

Answer 1

load the dump in windbg and do !analyze -v if the symbols are good you should get an
accurate result including source lines

taking the return address and feeding it to uf is not going to get anything useful

uf does a control flow and if the return address points to a jmp to some random function in some random dll uf will simply dump the random function .

return address is pushed into the stack by the callee

if you do ub <return address> you can find the callee

should be something like call blah <0x.......>

now do an uf on the callee uf <0x........> to see what the call does
most probably it looks like it is executing an unhandled exception

Answer 2

This is not exactly an answer yet, but too long for a comment. In the order of appearance:

"when the windows error reporting dialog was open."

Taking a crash dump while Windows Error Reporting is already working with the process does not work very well. Instead, run the application using ProcDump directly:

procdump -e -ma -x c:\MyDump.dmp C:\PathTo\MyApplication.exe Arguments

ProcDump will then generate a crash dump when the exception occurs. If that does not work well for you, let Windows Error Reporting save the dump on disk. See Collecting User Mode dumps for the Registry settings to do so. Be sure to set DumpType to 2 for .NET.

"I cannot find any .NET modules loaded"

How did you check for .NET modules? You can list all modules using lm and see if you find some DLLs you know.

"Setup the symbol path with ..."

That's ok. If you don't want to type all that HTTP stuff, you can also use

.symfix C:\Symbols
.sympath+ C:\PathToMatchingPDBs\ForTheProgram

"~* k outputs KERNELBASE!UnhandledExceptionFilter+0x1f1"

That could be the reason why you don't get much information about the original exception: the exception is just getting handled by Windows Error Reporting.

"Unable to find module ‘clr’"

Are you sure it's a .NET 4 application? Try .loadby sos mscorwks to see whether it's .NET 2. If that won't help either, check lm m ms* for other .NET-like assemblies.