CODEWITHSUNDEEP
dockergrafanaprometheus
HypeWolf
HypeWolf

Reputation: 850

How to make prometheus work with grafana in Docker?

The main goal is to link prometheus as a backend in grafana, but entering http://localhost:9090 as the url in grafana returns HTTP Error Bad Gateway

I started a prometheus docker image but it's not listening on port 9090 on IPv4.

netstat -ntulp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      15895/nginx: master 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3190/sshd           
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      24970/postmaster    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3148/master         
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      15895/nginx: master 
tcp6       0      0 :::9100                 :::*                    LISTEN      16652/node_exporter 
tcp6       0      0 :::80                   :::*                    LISTEN      15895/nginx: master 
tcp6       0      0 :::22                   :::*                    LISTEN      3190/sshd           
tcp6       0      0 :::3000                 :::*                    LISTEN      28436/docker-proxy  
tcp6       0      0 ::1:5432                :::*                    LISTEN      24970/postmaster    
tcp6       0      0 ::1:25                  :::*                    LISTEN      3148/master         
tcp6       0      0 :::9090                 :::*                    LISTEN      31648/docker-proxy  
udp        0      0 0.0.0.0:68              0.0.0.0:*                           2806/dhclient       
udp        0      0 127.0.0.1:323           0.0.0.0:*                           1639/chronyd        
udp6       0      0 ::1:323                 :::*                                1639/chronyd

This is my docker command:

docker run -d -p 9090:9090 --name prometheus -v /etc/prometheus.yml:/etc/prometheus/prometheus.yml -v /mnt/vol-0001/prometheus_data/:/etc/prometheus/data prom/prometheus --log.level=debug

I used -p 9090:9090 and -p 0.0.0.0:9090 with same results

docker logs prometheus returns:

level=info ts=2018-12-19T21:07:59.332452641Z caller=main.go:243 msg="Starting Prometheus" version="(version=2.6.0, branch=HEAD, revision=dbd1d58c894775c0788470944b818cc724f550fb)"
level=info ts=2018-12-19T21:07:59.332554622Z caller=main.go:244 build_context="(go=go1.11.3, user=root@bf5760470f13, date=20181217-15:14:46)"
level=info ts=2018-12-19T21:07:59.332584047Z caller=main.go:245 host_details="(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 9dd3a9318064 (none))"
level=info ts=2018-12-19T21:07:59.332610547Z caller=main.go:246 fd_limits="(soft=65536, hard=65536)"
level=info ts=2018-12-19T21:07:59.332631287Z caller=main.go:247 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2018-12-19T21:07:59.334232116Z caller=main.go:561 msg="Starting TSDB ..."
level=info ts=2018-12-19T21:07:59.334671887Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1545204931123 maxt=1545220800000 ulid=01CZ3PHTVQQTW7Q122X7Y15WV4
level=info ts=2018-12-19T21:07:59.334756938Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1545242400000 maxt=1545249600000 ulid=01CZ44997810VTYP3GV0KJXXN1
level=info ts=2018-12-19T21:07:59.334819198Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1545220800000 maxt=1545242400000 ulid=01CZ4499ASP4RG8BPR8PE5WAKY
level=info ts=2018-12-19T21:07:59.346244745Z caller=web.go:429 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-12-19T21:07:59.461554488Z caller=main.go:571 msg="TSDB started"
level=info ts=2018-12-19T21:07:59.461625871Z caller=main.go:631 msg="Loading configuration file" filename=prometheus.yml
level=debug ts=2018-12-19T21:07:59.462558422Z caller=manager.go:213 component="discovery manager scrape" msg="Starting provider" provider=string/0 subs=[prometheus]
level=info ts=2018-12-19T21:07:59.462601563Z caller=main.go:657 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-12-19T21:07:59.462615458Z caller=main.go:530 msg="Server is ready to receive web requests."
level=debug ts=2018-12-19T21:07:59.462669264Z caller=manager.go:231 component="discovery manager scrape" msg="discoverer channel closed" provider=string/0

I also tried disabling the firewall to make sure it wasn't the cause of this headache.

I'm no docker/kubernetes expert, you help is appreciate.

Upvotes: 3

Views: 3094

Answers (1)

prometherion
prometherion

Reputation: 2299

The localhost you're referring in Grafana Datasource input it's the Grafana container itself since Grafana internally resolves localhost as 127.0.0.1: probably since you're using the GUI you was expecting that the queries were issued via AJAX/frontend calls but nope, it's all backed by the backend.

Let orchestrate containers using even Docker Compose with services that connect container using Networks:

# docker-compose.yaml
version: "3"

services:
  grafana:
    image: grafana/grafana:5.4.1
    ports:
    - 3000:3000
  prometheus:
    image: prom/prometheus:v2.5.0

After docker-compose up -d you can visit your Docker Machine IP (or localhost if running Docker for Mac) at port :3000 and then set the Prometheus data source URL to http://prometheus:9090 and it will work!

Upvotes: 6

Related Questions