CODEWITHSUNDEEP
.netmetricsopen-telemetryexporter
MetalHexx
MetalHexx

Reputation: 41

OpenTelemetry .Net Application - Metrics Collected Slowly by Collector

I am running a simple POC for using open telemetry in .net 6 web api. I am trying to use the Otlp exporter in the .net app to send metrics to a collector. Prometheus is scraping the endpoint that the collector exposes just fine.

However, the initial export of metrics takes a long time. It seems to send metrics every 60 seconds once it gets going. I really would prefer to set it to a more reasonable 5-15 seconds or so to make it demo better.

I tried the simple exporter with no luck. I tried to set the batch importer with a very low MaxQueueSize, ScheduledDelayMilliseconds and ExporterTimeoutMilliseconds.

When enabling a ConsoleExporter on the .net application side, the metrics seem to come through the expected default of 10 seconds. So it may be an issue with the collector doing some sort of accumulation before rendering an export on the metrics endpoint? I'm not using a batch processor in the collector, so I am not sure why it would do that.

I also didn't have any slowness issues when I used the PrometheusExporter / scrape endpoint on the API itself. It's only when I try to export metrics with the Otel Exporter.

What is interesting is that I also have the OTLP exporter configured for tracing and I get all my results into the collector / jaeger almost instantly.

I am currently using the latest Open Telemetry stable packages (1.32.0). I am using the following docker image for the collector: otel/opentelemetry-collector-contrib:0.42.0

Code available here: https://github.com/MetalHexx/OpenTelemetryDotNetPoc

Abbreviated startup code:

services.AddOpenTelemetryMetrics(builder =>
{
     builder
         .AddMeter(App_Source)
         .SetResourceBuilder(resource!)
         .AddAspNetCoreInstrumentation()
         .AddOtlpExporter(options =>
         {
             options.Endpoint = new Uri("http://poc-collector:4319/v1/metrics");
             options.Protocol = OtlpExportProtocol.HttpProtobuf;
             options.ExportProcessorType = ExportProcessorType.Simple;
         });
});

Abbreviated Collector config:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4319

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: poc 

  logging:
    loglevel: debug

service:
  telemetry:
    logs:
      level: "debug"
  extensions: []
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [logging, prometheus]

Upvotes: 2

Views: 2300

Answers (1)

MetalHexx
MetalHexx

Reputation: 41

I found the solution. According to the specification, indeed the default is 60 seconds.
The solution is to use the OTEL_METRIC_EXPORT_INTERVAL environment variable.

In the .net SDK you can specify it by adding the metricReaderOptions parameter, then you can set the number of milliseconds for the export interval:

.AddOtlpExporter((exporterOptions, metricReaderOptions) =>
{
     exporterOptions.Endpoint = new Uri("http://poc-collector:4319/v1/metrics"); 
     exporterOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
     exporterOptions.ExportProcessorType = ExportProcessorType.Simple;
metricReaderOptions.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 5000;

})

Credit goes to utpilla on Github: https://github.com/open-telemetry/opentelemetry-dotnet/issues/4026#issuecomment-1363411811

Upvotes: 2

Related Questions