cAdvisor does not show all container's uptime (Prometheus+cAdvisor+Grafana)

Question

Environment:

• Linux (Redhat7)
• Deployed docker (cAdvisor, Prometheus, Grafana)
• cAdvisor collect the metrics > Pass to Prometheus > Display with Grafana
• Apache reverse proxy is in the environment (Therefore no direct connection with specific ports)

Issue:

• cAdvisor does not show all container's uptime
• Grafana does not show Prometheus and Grafana container's uptime
• Only displays cAdvisor container's uptime

What I have (Issue):

What I want to have:

Setting in Prometheus:

Command to run cAdvisor:

sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/dev/disk/:/dev/disk:ro --publish=8080:8080 --name=cadvisor --detach=true --privileged=true --volume=/cgroup:/cgroup:ro --network=docker8443 --ip=172.28.5.201 google/cadvisor:latest

Questions:

• Is something missing in Prometheus target to show all container's uptime?
• Is something missing in cAdvisor setup?
• Is query for the graph incorrect? (I have tried following, but it shows "No data point")

Answer 1

On top of Yunlong's answer: add --volume=/sys:/sys:ro You will also need to run following under shell:

$ mount -o remount,rw '/sys/fs/cgroup'

$ ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu

Otherwise your cAdvisor crashes immediately after container is up.

Answer 2

I think the issue is about cadvisor. You can try check the cadvisor metrics endpoint: http://172.28.5.201:8080/metrics to make sure the cadvisor return the metrics samples:

example:

# HELP container_start_time_seconds Start time of the container since unix epoch in seconds.
# TYPE container_start_time_seconds gauge
container_start_time_seconds{id="/",image="",name=""} 1.525939343e+09
container_start_time_seconds{id="/docker",image="",name=""} 1.526006565e+09
container_start_time_seconds{id="/docker/d4b87911bd0842ee1d6969e6a05aa3d36a48a801184faf14e1b23169e056da92",image="busybox",name="trusting_bassi"}