CODEWITHSUNDEEP
azure-application-insightscost-management
Darren Neimke
Darren Neimke

Reputation: 268

Why am I seeing a large spike in Data ingestion from performance counter data?

From looking at my Cost Management reports for Application Insights, I've noticed a large spike in data ingestion related to Performance Counters.

Cost Management Reporting

Link to larger image

However, I'm not seeing any significant correlation in the number of requests spiking for the same period.

Request Metrics Report

Link to larger image

What can I do to understand the root cause of this issue?

Additional Information

After some debugging, I was able to get to the bottom of this.

The usage spiked on Sep 7 and Sep 8 and then tapered off on the 9th and 10th.

The change that I made on the 6th Sept was to upgrade Microsoft.ApplicationInsights.AspNetCore from version 2.6.1 to version 2.7.1. Version 2.7.1 has ILogger integration baked in.

So, what I believe happened is that, after deploying the upgraded version of Microsoft.ApplicationInsights.AspNetCore, I may have had the Logging verbosity turned up too high for the performance counter telemetry data and subsequently changed it a couple of days later when I noticed it.

I hope this helps someone else who may run into this issue!

Upvotes: 0

Views: 811

Answers (2)

alv
alv

Reputation: 1097

There is now a cleaner way to achieve the disabling of the PerformanceCounterModule (and other modules which cause excessive/unneeded logging);

services.AddApplicationInsightsTelemetry(options =>
{
    options.EnablePerformanceCounterCollectionModule = false;
    options.InstrumentationKey = configuration["ApplicationInsights:InstrumentationKey"];
});

Upvotes: 1

cijothomas
cijothomas

Reputation: 3196

Application Insights 2.7.1 has enabled ILogger captured by default, but it only captures Warning or above log messages. So unless you application is generating a lot of Warning or above level Ilogger logs, this should not result in a big spike in usage. This behavior can be changed to filter logs further if too much logs are reported. https://learn.microsoft.com/en-us/azure/azure-monitor/app/ilogger#control-logging-level From the 1st screenshot you shared, it looks like performance counter is the only type which spiked - ilogger integration cannot explain this spike, as it only reports logs.

More logical explanation is the PerformanceCounter module itself which was NOT supported in versions earlier than 2.7.1. You can remove performance counter collection with the following snipped in ConfigureServices() method in startup.cs

using Microsoft.ApplicationInsights.DependencyCollector; using Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector;

public void ConfigureServices(IServiceCollection services)
{
    services.AddApplicationInsightsTelemetry();

    // The following removes PerformanceCollectorModule to disable perf-counter collection.
    var performanceCounterService = services.FirstOrDefault<ServiceDescriptor>(t => t.ImplementationType == typeof(PerformanceCollectorModule));
    if (performanceCounterService != null)
    {
     services.Remove(performanceCounterService);
    }
}

Upvotes: 1

Related Questions