CODEWITHSUNDEEP
dockernetwork-programmingiptablessocat
The Fool
The Fool

Reputation: 20457

scrape docker metrics from inside container

I am trying to scrape docker metrics from inside a container. I have seen the creator of Prometheus doing it this way see video here. I am not able to reproduce it.

Configuration

I have exposed docker metrics page via json config.

{
  "metrics-addr" : "172.17.0.1:4999",
  "experimental" : true
}

I have also tried with 0.0.0.0. In any case I am able to curl from the host machine itself with no issues.

$ curl 172.17.0.1:4999/metrics | more

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
...

From inside container

However, I am not able to get the same metrics page from inside a container. I have tried to basi/socat image, the same one from the video. I have also tried a handful of other ways like using curl from inside the container and so on.

The image does essentially just one thing

ENV IN="172.17.0.1:4999" \
    OUT="4999"

ENTRYPOINT socat -d -d TCP-L:$OUT,fork TCP:$IN

Start Socat Container

docker run --rm -p 4998:4999 basi/socat
2021/04/02 16:27:47 socat[8] N listening on AF=2 0.0.0.0:4999

curl the Container published port

curl localhost:4998/metrics

Connection just hangs

The socat images recognizes the connection, but it hangs for 1-2 minutes and times out.

2021/04/02 16:29:53 socat[8] N accepting connection from AF=2 172.17.0.1:57678 on AF=2 172.17.0.2:4999
2021/04/02 16:29:53 socat[8] N forked off child process 9
2021/04/02 16:29:53 socat[8] N listening on AF=2 0.0.0.0:4999
2021/04/02 16:29:53 socat[9] N opening connection to AF=2 172.17.0.1:4999
2021/04/02 16:32:04 socat[9] E connect(5, AF=2 172.17.0.1:4999, 16): Operation timed out
2021/04/02 16:32:04 socat[9] N exit(1)
2021/04/02 16:32:04 socat[8] N childdied(): handling signal 17
2021/04/02 16:32:04 socat[8] W waitpid(): child 9 exited with status 1

curl: (52) Empty reply from server

Firewall

I am slowly wondering if it could be something in my iptables rules. I have posted below INPUT and DOCKER-USER chain. Those are the only ones I have changed myself.

sudo iptables -L -n -v
Chain INPUT (policy DROP 32060 packets, 1810K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 716K 1484M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
 599K 1069M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   16  3138 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   12   624 DROP       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           
74921  139M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Upvotes: 0

Views: 666

Answers (1)

The Fool
The Fool

Reputation: 20457

So turns out issues was indeed iptables. The default policy for the INPUT chain in this case is DROP. Since the container is now trying to reach the host system itself, its traffic will come through the input chain.

So I added 2 simple rules for now, that will allow all traffic in the INPUT chain from docker0 and docker_gwbridge.

sudo iptables -A INPUT -i docker0 -j ACCEPT
sudo iptables -A INPUT -i docker_gwbridge -j ACCEPT

After that I am finally able to query docker metrics from inside the container.

docker run --rm curlimages/curl -I -m 10 172.17.0.1:4999/metrics
HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Sat, 03 Apr 2021 10:53:12 GMT

Upvotes: 1

Related Questions