CODEWITHSUNDEEP
.netopen-telemetrysystem.diagnosticsotel
luke77
luke77

Reputation: 232

.net 8 - Migration from AppMetrics to Meter - OpenTelemetry => High CPU

We are monitoring the duration of rate of error of some methods in our code. To do that, we use an AOP Attribute (using Postsharp) and AppMetrics to store the metrics in a Postgresql server.

Everything is working smoothly.

We decided to migrate from AppMetrics to OpenTelemetry, and replaced the calls to AppMetrics by calling Counter and Histogram of System.Diagnostics.

The system monitors about 300 hundred methods, and the tags associated with metric contains a client code (we are a multi-tenant solution).

In theory, we could have 21 000 different series handled by the system. The attribute is placed on critical methods (potentially called a lot of times per second).

Since we have made this change, our process use 100% CPU of the server.

Removing the Meter / Histogram code make it back to a normal consumption.

Does someone have any idea of what could happen ? Does System.Diagnostics not suitable for this kind of metrics (and this volume) ?

EDIT

The AOP attribute is instanciated once per method monitored (so about 300 times).

private static readonly Histogram<double> DurationHistogram = Meter.CreateHistogram<double>("measure_execution_time_duration", "ms", "Duration of method execution in milliseconds"); 
private static readonly Counter<long> SuccessCounter = Meter.CreateCounter<long>("measure_execution_time_success", "calls", "Total number of successful calls"); 
private static readonly Counter<long> ErrorCounter = Meter.CreateCounter<long>("measure_execution_time_error", "calls", "Total number of error calls");

Then we add a value like this :

var tags = BuildTags(args); 
SuccessCounter.Add(1, tags);

The tags contains the following fields

Upvotes: 0

Views: 46

Answers (0)

Related Questions