CODEWITHSUNDEEP
dockermonitoringprometheus
louis
louis

Reputation: 93

Can't collect docker metrics using prometheus

I want to monitor the Docker engine. I am following "Collect Docker metrics with Prometheus" to configure the Docker daemon for metrics which can be later collected and analyzed with Prometheus.

On my PC (OS version is CentOS 7.4, Docker version is 17.12.0.ce), I have pasted the following to /etc/docker/daemon.json

    {
      "metrics-addr" : "0.0.0.0:9323",
      "experimental" : true
    }

Then, starting the Prometheus instance, I found the connection between Prometheus and Docker daemon is failed.

Error log

Get http://localhost:9323/metrics: dial tcp 127.0.0.1:9323: connect: connection refused

What can I do next?

Upvotes: 6

Views: 12036

Answers (5)

U880D
U880D

Reputation: 11999

In respect to your question

What can I do next?

after changing the configuration in /etc/docker/daemon.json it will be necessary to

systemctl daemon-reload
systemctl restart docker

You may check after this if the port 9323 is in state LISTEN

lsof -Pi TCP -a -c dockerd

As you mentioned in your comment that

curl http://$(hostname):9323/metrics

is working properly on your local host, it indicates a problem with(in) your network.

As the Prometheus service is usually running somewhere else you may check if there is a connection from the remote machine to your Docker host. For this you could use something like

root@prometheusHost:/# nc -vz dockerHost 9323

It will give you a hint if the connection is refused, e.g. by a firewall.

Upvotes: 5

JrCaspian
JrCaspian

Reputation: 327

If you run Prometheus from a container, you can consider the below address:

host.docker.internal:9323

Add this to your prometheus.yml inside your Prometheus container.

Upvotes: 1

Ilia Dimchev
Ilia Dimchev

Reputation: 21

I've run into the same issue, here's the fix: $ sudo vi /etc/hosts

0.0.0.0 <hostname> host

0.0.0.0 can be substituted with the advertised address in Docker Swarm

Upvotes: 0

zabumba
zabumba

Reputation: 12402

Configure the Docker daemon

NOTE: 0.0.0.0 i.e. all IP addresses on the local machine

/etc/docker/daemon.json

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Open firewall port 9323 --permanently

$ sudo firewall-cmd --zone=public --permanent --add-port=9323/tcp
success

A few checks

Check, remember to list --permanent configuration, or it won't show

$ sudo firewall-cmd --zone=public --permanent --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client http https ssh
  ports: 9323/tcp                            <= OPEN
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 


$ sudo lsof -Pi TCP -a -c dockerd
COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
dockerd-c 28157 root   14u  IPv6 30732963      0t0  TCP *:9323 (LISTEN)
...

Check connection to localhost

nc -vz localhost 9323
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to ::1:9323.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check connection to host IP e.g. 10.223.37.14

$ nc -vz 10.223.37.14 9323
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.223.37.14:9323.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check the metrics acccess using curl

curl http://localhost:9323/metrics

or using the IP from another machine

curl http://10.223.37.14:9323/metrics

You will get a connection refused if the port is not open

curl http://10.223.37.14:9323/metrics
curl: (7) Failed to connect to 10.223.37.14 port 9323: Connection refused

but once it's open, you will be able to see the metrics

$ curl http://10.223.37.14:9323/metrics
# HELP engine_daemon_container_actions_seconds The number of seconds it takes to process each container action
# TYPE engine_daemon_container_actions_seconds histogram
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.005"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.01"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.025"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.05"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.1"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.25"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.5"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="1"} 1
...

Upvotes: 2

Misho M. Petkovic
Misho M. Petkovic

Reputation: 364

Get docker ip addres

ip addr show docker0

Enter docker ip addres in prometherus.yml configuration

    static_configs:
  - targets: ['172.17.0.1:9323']

enter image description here

Upvotes: 6

Related Questions