CODEWITHSUNDEEP
goprometheusmetricsopen-telemetry
miroshnik
miroshnik

Reputation: 163

How to distinguish metrics from different services

I'm playing with OpenTelemetry and have such a setup:

Golang, docker-compose, 3 services, 1 standalone open-telemetry collector, 1 Prometheus.

enter image description here

I collect some system metrics to a standalone open-telemetry collector. These metrics are collected from 3 different services and metrics have identical names. Then, Prometheus gets the data from the open-telemetry collector. The problem is that I can't distinguish metrics from different services in Prometheus because all of the metrics have the same "instance" value, which is equal to the open-telemetry-collector's host.

I know that I can add a label with a service's name to the metric record and then distinguish the metrics by the label, but I'm searching for another solution because it is not always possible to add the label to each metric. Maybe, something like http-middleware, but for metrics, or maybe something on an infrastructure level.

Services are written with Golang, but I will be glad to see the solution in any other language.

otel-collector-config:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  prometheus:
    endpoint: otel-collector:8889
    const_labels:
      label1: value1
    send_timestamps: true
    metric_expiration: 5m

processors:
  batch:

service:
  pipelines:
    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ prometheus ]

Prometheus config:

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 5s
    static_configs:
      - targets: ['otel-collector:8889']

docker-compose:

version: "3.9"
services:

  service1:
    build:
      context: ./service1
      network: host
    environment:
      - TELEMETRY_COLLECTOR_ADDR=otel-collector:55681
    ports:
      - "8094:8080"
    expose:
      - "8080"

  service2:
    build:
      context: ./service2
      network: host
    environment:
      - TELEMETRY_COLLECTOR_ADDR=otel-collector:55681
    ports:
      - "8095:8080"
    expose:
      - "8080"

  service3:
    build:
      context: ./service3
      network: host
    environment:
      - TELEMETRY_COLLECTOR_ADDR=otel-collector:55681
    expose:
      - "8080"
    ports:
      - "8096:8080"

  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  otel-collector:
    image: otel/opentelemetry-collector:0.23.0
    command: [ "--config=/etc/otel-collector-config.yaml" ]
    expose:
      - "55681" # HTTP otel receiver
      - "8889" # Prometheus exporter metrics
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml

Update 1. I found that some new parameters were added to exporter-config https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper . One of them is what suitable for me: resource_to_telemetry_conversion. But as I see prometheusexporter and prometheusremotewriteexporter don't support that field in the config.

Upvotes: 2

Views: 1992

Answers (1)

Edinardo Potrich
Edinardo Potrich

Reputation: 36

The resource_to_telemetry_conversion that you mentioned is part of prometheusexporter since version 0.26.0 (issue #2498) and will add the service_name label based on the agent settings to distinguish metrics from different services.

Upvotes: 2

Related Questions