CODEWITHSUNDEEP
dockerdocker-composeprometheus
Martijn Hols
Martijn Hols

Reputation: 1598

With Prometheus how to monitor a scaled Docker service where each instance serves its own /metrics?

I have a Prometheus setup that monitors metrics exposed by my own services. This works fine for a single instance, but once I start scaling them, Prometheus gets completely confused and starts tracking incorrect values.

All services are running on a single node, through docker-compose.

This is the job in the scrape_configs:

  - job_name: 'wowanalyzer'
    static_configs:
    - targets: ['prod:8000']

Each instance of prod tracks metrics in its memory and serves it at /metrics. I'm guessing Prometheus picks a random container each time it scraps which leads to the huge increase in counts recorded, building up over time. Instead I'd like Prometheus to read /metrics on all instances simultaneously, regardless of the amount of instances active at that time.

Upvotes: 7

Views: 1650

Answers (3)

Alex
Alex

Reputation: 2126

You can use DNS service discovery feature. For example:

docker-compose.yml:

version: "3"
services:
  myapp:
    image: appimage:v1
    restart: always
    networks:
      - back
  prometheus:
    image: "prom/prometheus:v2.32.1"
    container_name: "prometheus"
    restart: "always"
    ports: [ "9090:9090" ]
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "prometheus_data:/prometheus"
    networks:
      - back

prometheus.yml sample:

global:
  scrape_interval: 15s
  evaluation_interval: 60s
scrape_configs:
  - job_name: 'monitoringjob'
    dns_sd_configs:
      - names: [ 'myapp' ] <-- service name from docker-compose 
        type: 'A'
        port: 8080
    metrics_path: '/actuator/prometheus'

You can check your DNS records using nslookup util from any container in this network:

docker exec -it myapp bash
bash-4.2# yum install bind-utils
bash-4.2# nslookup myapp
Server:     127.0.0.11
Address:    127.0.0.11#53

Non-authoritative answer:
Name:   myapp
Address: 172.22.0.2
Name:   myapp
Address: 172.22.0.7

Upvotes: 2

Fraak
Fraak

Reputation: 218

docker-gen (https://github.com/jwilder/docker-gen) was developed for this purpose.

You would need to create a sidecart container running docker-gen that generates a new set of targets.

If I remember well the host names generated are prod_1, prod_2, prod_X, etc.

Upvotes: 2

J&#250;lio Falbo
J&#250;lio Falbo

Reputation: 1926

I tried a lot to find something to help us with this issue but it looks an unsolved issue.

So, I decided to create this tool that helps us with this service-discovery.

https://github.com/juliofalbo/docker-compose-prometheus-service-discovery

Feel free to contribute and open issues!

Upvotes: 1

Related Questions