Opentelemetry with Prometheus - Metric Data points / scrapes missing

Question

Our Set-up in Kubernetes is as follows:

Opentelemetry agent (both java and python used and metrics from both have the same issue)

Opentelemetry-collector - version 0.77.0 (we previously were on 0.72.0 and that had the same problem) It's using the otlp receiver, like this:

    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

It has the following processors configured:

  processors:
    memory_limiter:
      limit_mib: 4000
      spike_limit_mib: 800
      check_interval: 1s
    batch:
      send_batch_size: 10000
      send_batch_max_size: 11000
      timeout: 10s
    #Delete unnecessary attributes from our metrics
    resource:
      attributes:
        - key: telemetry.sdk.name
          action: delete
        - key: telemetry.sdk.version
          action: delete
        - key: telemetry.sdk.language
          action: delete
        - key: telemetry.auto.version
          action: delete
        - key: container.id
          action: delete
        - key: process.command.args
          action: delete
    k8sattributes/default:

Previously we had none of these processors configured and the problem was exactly the same

the opentelemetry collector has the prometheus exporter set like this:

  exporters:
    prometheus:
      endpoint: "0.0.0.0:9464"
      resource_to_telemetry_conversion:
        enabled: true
      enable_open_metrics: true
      namespace: prometheus

and then servicemonitors configured like this:

serviceMonitor:
  enabled: true
  metricsEndpoints:
    - port: metrics
      interval: 15s
  prometheusMetricsEndpoints:
    - port: app-metrics
      interval: 15s
      honorLabels: true

Then finally we have prometheus which scrapes for these metrics based on those servicemonitors and Grafana to visualise everything.

The problem we have is that for some of our metrics we randomly see that datapoints are missing... we usually get a datapoint every 15 seconds but sometimes it's just not there, here are some examples:

for the logback_events_total metric we see many empty datapoints:

and another example with http_server_requests_milliseconds_count this time zoomed in a bit more to make it more clear showing points only, no lines:

in this one you can clearly see we get a point every 15 seconds but sometimes we miss one or two of these points.

I have looked at the metrics for machine_scrape_error and container_scrape_error and as you can see there are no dropped data points but more importantly no scrape errors being reported

This problem seems to only happen for Counter based metrics... for others we don't seem to get this problem at all (e.g jvm_threads_live). Our Opentelemetry collector pod is only using roughly 10% CPU and 20% memory Our prometheus is scaled to 6 pods with resource usage well under 50% of the requested amount.

I am running out of ideas about what could cause this, I understand that a few dropped scrapes is normal but this seems like more than would be acceptable. We have tried turning on debug logs but haven't found anything useful in the logs. I assume these data points could be going missing at different points along the way (agent, collector or prometheus). Does anyone have any advice about how to start tracking this down?

Adding another graph as it was mentioned in the comments: scrape duration seconds plotted alongside another metric that was dropping data points: scrape duration seconds is the green line with no gaps

Edit: 30th May 2023 Still having the same issue, after enabling error logs on the otel collector we are seeing lines like these:

* collected metric prometheus_http_server_duration_milliseconds label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: histogram: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: >  has help "The duration of the inbound HTTP request" but should have "measures the duration of the inbound HTTP request"
* collected metric prometheus_jvm_threads_states label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: gauge:  has help "The current number of threads having NEW state" but should have "The current number of threads"
* collected metric prometheus_http_server_duration_milliseconds label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: label: histogram: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: bucket: >  has help "The duration of the inbound HTTP request" but should have "measures the duration of the inbound HTTP request"

This would suggest a mis-match between the description of the metric being received and the actual spec of that metric?

but, it seems strange that this would only be for some data points in a metric and not every one... does this help to narrow down the issue

Opentelemetry with Prometheus - Metric Data points / scrapes missing

Answers (1)

Related Questions