Missing Stack Frames from ClrStackWalk Event

Question

I try to use TraceProcessing Library to get from managed exceptions which have an associated Clr Stackwalk event the stack trace. In principle it should be pretty easy by parsing the event and getting the method addresses.

using Microsoft.Windows.EventTracing;
using Microsoft.Windows.EventTracing.Events;
using System;
using System.Collections.Generic;
using System.Linq;

namespace TraceProcessingStackDecoding
{
    class Program
    {
        static void Main(string[] args)
        {
            string etlFile = args[0];
            using ITraceProcessor processor = TraceProcessor.Create(etlFile, new TraceProcessorSettings
            {
                AllowLostEvents = true,
            });

            IPendingResult<IGenericEventDataSource> genericEvents = processor.UseGenericEvents();
            processor.Process();


            const int ClrStackWalkEventId = 82;
            const string DotNetRuntimeProviderName = "Microsoft-Windows-DotNETRuntime";

            foreach (IGenericEvent clrStackWalk in genericEvents.Result.Events.Where( x=> x.ProviderName == DotNetRuntimeProviderName && x.Id == ClrStackWalkEventId))
            {
                IReadOnlyList<Address> stackAddresses = clrStackWalk.Fields["Stack"].AsAddressList;
                uint frameCount = clrStackWalk.Fields["FrameCount"].AsUInt32;

                if( stackAddresses.Count != frameCount)
                {
                    Console.WriteLine($"Error: Address List has only {stackAddresses.Count} entries but expected were {frameCount} entries!");
                }
            }
        }
    }
}

But when I do this I find that almost all Stack frames are missing. I always get 2. If I am not mistaken it should return all data until the end of the event.

Error: Address List has only 2 entries but expected were 34 entries!
Error: Address List has only 2 entries but expected were 35 entries!
Error: Address List has only 2 entries but expected were 35 entries!
Error: Address List has only 2 entries but expected were 35 entries!
Error: Address List has only 2 entries but expected were 36 entries!
Error: Address List has only 2 entries but expected were 37 entries!
Error: Address List has only 2 entries but expected were 64 entries!
Error: Address List has only 2 entries but expected were 30 entries!
Error: Address List has only 2 entries but expected were 77 entries!
Error: Address List has only 2 entries but expected were 77 entries!
Error: Address List has only 2 entries but expected were 31 entries!
Error: Address List has only 2 entries but expected were 77 entries!

The Clr Stackwalk event manifest defines it as this:

            <template tid="ClrStackWalk">
                <data name="ClrInstanceID" inType="win:UInt16"/>
                <data name="Reserved1" inType="win:UInt8"/>
                <data name="Reserved2" inType="win:UInt8"/>
                <data name="FrameCount" inType="win:UInt32"/>
                <data name="Stack" count="2" inType="win:Pointer"/>
            </template>

The problem might be the count property which you take serious. But that is not how the event is actually logged and addresses lists are almost 100% dynamic list of stacks which do not have a fixed count. The best would be to return all data until the end of the event as address list if it is the last item in the manifested event.

Since I cannot access the raw event I have only a nice type safe albeit useless wrapper which makes it impossible to get the stack frames for .NET Stackwalk events. Besides this does TraceProcessing support also JITed code when I try to lookup symbols? At API level I can only find a method at Image Level which will make it impossible to decode JITed code? But since TraceProcessing can decode JITed call stacks I think there might be something missing at API level.

foreach (Address stackAdr in stackAddresses)
{
    foreach (var image in ev.Process.Images)
    {
        var range = image.AddressRange;
        if (  ( (range.BaseAddress < range.LimitAddress) && (stackAdr > range.BaseAddress && stackAdr < range.LimitAddress)) ||
              ( (range.BaseAddress > range.LimitAddress) && (stackAdr < range.BaseAddress && stackAdr > range.LimitAddress)) )
        {
            IStackSymbol stackSymbol = image.GetSymbol(stackAdr);
            Console.WriteLine(stackSymbol?.FunctionName);
        }
    }
}

Will this approach work with JITed code out of the box or do I need to decode all JIT events manually?

Answer 1

If the manifest and the event payload do not match, parsing them yourself may be the best workaround. (I'm glad that worked for you.)

The best option would probably be to get the manifest fixed - does that team have a GitHub repository where you can file a bug? Maybe the folks at the following repository would know? https://github.com/dotnet/diagnostics

Answer 2

I have got it working in some hacky way by parsing all events and then get the raw data of the events and store the addresses on my own. Later when I parse the generic events I can map the corresponding event by its time stamp and resolve the methods. The managed methods nicely show up there also for JITed code.

    ...    
     processor.Use(ProcessRawEvents);
    ...

    List<StackEvent> StackEvents = new List<StackEvent>();
    
    class StackEvent
    {
        public TraceTimestamp TimeStamp;
        public IReadOnlyList<Address> Stack;
    }
    
    bool myNeedsStack = false;
    
    
void ProcessRawEvents(EventContext eventContext)
{
    TraceEvent ev = eventContext.Event;

    if (ev.ProviderId == Constants.DotNetRuntimeGuid)
    {
        if (ev.Id == Constants.ExceptionEventId)
        {
            myNeedsStack = true;
        }

        // potentially every exception event is followed by a stackwalk event
        if (myNeedsStack && ev.Id == Constants.ClrStackWalkEventId)
        {
            myNeedsStack = false;

            StackEvent stackEv = new StackEvent()
            {
                TimeStamp = ev.Timestamp,
            };

            ReadOnlySpan<byte> frameData = ev.Data.Slice(8);
            List<Address> addresses = new List<Address>();
            stackEv.Stack = addresses;

            if (ev.Is32Bit)
            {
                ReadOnlySpan<int> ints  = MemoryMarshal.Cast<byte, int>(frameData);
                    
                foreach(var intAdr in ints)
                {
                    addresses.Add(new Address(intAdr));
                }
            }
            else
            {
                ReadOnlySpan<long> longs = MemoryMarshal.Cast<byte, long>(frameData);
                foreach(var longAdr in longs)
                {
                    addresses.Add(new Address(longAdr));
                }
            }

            StackEvents.Add(stackEv);
        }
    }

}