Application insights, and it's maximum storage ability on telemetry

Question

I have a middleware telemetry handler, that has a method that awaits the execution of a request, and then tries to store some key data values from the response body into custom dimensions fields in application insights, so I can use graphana and potentially other 3rd party products to analyse my reponses.

public class ResponseBodyHandler : IResponseBodyHandler
{
    private readonly ITelemetryPropertyHandler _telemetryPropertyHandler = new TelemetryPropertyHandler();
    
    public void TransformResponseBodyDataToTelemetryData(RequestTelemetry requestTelemetry, string responseBody)
    {
        SuccessResponse<List<Booking>> result = null;
        try
        {
            result = JsonConvert.DeserializeObject<SuccessResponse<List<Booking>>>(responseBody);
        }
        catch (Exception e)
        {
            Log.Error("Telemetry response handler, failure to deserialize response body: " + e.Message);
            return;
        }

        _telemetryPropertyHandler.CreateTelemetryProperties(requestTelemetry, result);
    }
}

public class TelemetryPropertyHandler : ITelemetryPropertyHandler
    {
        private readonly ILabelHandler _labelHandler = new LabelHandler();
        public void CreateTelemetryProperties(RequestTelemetry requestTelemetry, SuccessResponse<List<Booking>> result)
        {
            Header bookingHeader = result?.SuccessObject?.FirstOrDefault()?.BookingHeader;
            requestTelemetry?.Properties.Add("ResponseClientId", "" + bookingHeader?.ConsigneeNumber);
            
            Line line = bookingHeader?.Lines.FirstOrDefault();
            requestTelemetry?.Properties.Add("ResponseProductId", "" + line?.PurchaseProductID);
            requestTelemetry?.Properties.Add("ResponseCarrierId", "" + line?.SubCarrierID);

            _labelHandler.HandleLabel(requestTelemetry, bookingHeader);
            
            requestTelemetry?.Properties.Add("ResponseBody", JsonConvert.SerializeObject(result));
        }
    }

Now, inside: _labelHandler.HandleLabel(requestTelemetry, bookingHeader); It extracts an Image that is base64 encoded, chunks up the string in sizes of 8192 characters, and adds them to the Properties as: Image index 0 .. N (N being the total chunks)

I can debug and verify that the code works. However, on application insights, the entire "request" entry, is missing, not just the custom dimensions. I am assuming that this is due to a maximum size constraint, and I am likely trying to add more data than is "allowed", however, I can't for the life of me, find the documentation that enforces this restriction.

Can someone tell what rule I am breaking? so I can either truncate the image out, if it isn't possible to store that much data? Or if there is something else I am doing wrong?

I have validated, that my code works fine, as long as I truncate the data into a single Property, that of course only partially stores the Image. (Making said "feature" useless)

public class LabelHandler : ILabelHandler
{
    private readonly IBase64Splitter _base64Splitter = new Base64Splitter();
    
    public void HandleLabel(RequestTelemetry requestTelemetry, Header bookingHeader)
    {
        Label label = bookingHeader?.Labels.FirstOrDefault();
        IEnumerable<List<char>> splitBase64String = _base64Splitter.SplitList(label?.Base64.ToList());

        if (splitBase64String != null)
        {
            bool imageHandlingWorked = true;
            try
            {
                int index = 0;
                foreach (List<char> chunkOfImageString in splitBase64String)
                {
                    string dictionaryKey = $"Image index {index}";
                    string chunkData = new string(chunkOfImageString.ToArray());
                    requestTelemetry?.Properties.Add(dictionaryKey, chunkData);
                    index++;
                }
            }
            catch (Exception e)
            {
                imageHandlingWorked = false;
                Log.Error("Error trying to store label in chunks: " + e.Message);
            }
            
            if (imageHandlingWorked && label != null)
            {
                label.Base64 = "";
            }
        }
    }
}

The above code is responsible for adding the chunks to a requestTelemetry Property field

public class Base64Splitter : IBase64Splitter
    {
        private const int ChunkSize = 8192;

        public IEnumerable<List<T>> SplitList<T>(List<T> originalList)
        {
            for (var i = 0; i < originalList.Count; i += ChunkSize)
                yield return originalList.GetRange(i, Math.Min(ChunkSize, originalList.Count - i));
        }
    }

This is the specific method for creating a char list chunk of characters, that correspond to the application insights maximum size pr custom dimension field.

Here is an image of the truncated field being added, if I just limit myself to a single property, but truncate the base64 encoded value.

Answer 1

[I'm from Application Insights team]

You can find field limits documented here: https://learn.microsoft.com/en-us/azure/azure-monitor/app/data-model-request-telemetry

On ingestion side there is a limit of 64 * 1024 bytes for overall JSON payload (need to add it to documentation).

You're facing something different though - that custom dimensions are removed completely. Maybe SDK detects that 64kb is exceeded and "mitigates" it this way. Can you try to limit it to a little bit less than 64kb?