CODEWITHSUNDEEP
javaspringspring-bootdockergrafana
Alex_Pap
Alex_Pap

Reputation: 464

Spring + Prometheus + Grafana: Err reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused

Hello I have an app in Spring Boot and I am exposing some metrics on Prometheus. My next goal is to provide these metrics on Grafana in order to obtain some beautiful dashboards. I am using docker on WSL Ubuntu and typed the next commands for Prometheus and Grafana:

docker run -d --name=prometheus -p 9090:9090 -v /mnt/d/Projects/Msc-Thesis-Project/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Below I am giving you the Prometheus dashboard in my browser and as you can see, everything is up and running. My problem is in Grafana configuration where I have to configure Prometheus as Data Source.

Prometheus Dashboard

In the field URL I am providing the http://localhost:9090 but I am getting the following error:

Error reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused

I've searched everywhere and saw some workarounds that don't apply to me. To be specific I used the following: http://host.docker.internal:9090, http://server-ip:9090 and of course my system's IP address via the ipconfig command http://<ip_address>:9090. Nothing works!!!

I am not using docker-compose but just a prometheus.yml file which is as follows.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'Spring Boot Application input'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 2s
    scheme: http
    static_configs:
      - targets: ['192.168.1.233:8080']
        labels:
          application: "MSc Project Thesis"

Can you advise me something?

Grafana Dshboard

Upvotes: 5

Views: 21637

Answers (9)

Qudrat Ullah
Qudrat Ullah

Reputation: 51

When you open http://localhost:9090 in your browser, you're accessing your local machine on port 9090, where Prometheus is running. Since the browser runs on your host machine, it works as expected.

However, when Grafana, running inside a Docker container, tries to access http://localhost:9090, it does not refer to your local machine. Instead, localhost inside a Docker container refers to the container itself, meaning Grafana is looking for Prometheus inside its own container. Since Prometheus is running in a separate container, Grafana cannot find it this way.

The correct approach is to use the Docker service name instead of localhost. In Docker Compose, each service is assigned a hostname that matches its service name from docker-compose.yml. So, replacing http://localhost:9090 with http://prometheus:9090 works because Grafana can now find Prometheus within the Docker network.

Summary: http://localhost:9090 works in your browser because the browser runs on your local machine. http://localhost:9090 does not work inside Grafana’s container because localhost refers to the Grafana container, not the host machine. http://prometheus:9090 works because Docker provides internal DNS resolution, allowing containers to communicate using service names.
✅✅✅✅✅✅✅✅✅
Pro tip: When connecting services within Docker, always use service names instead of localhost.

Upvotes: 0

novice_coder99
novice_coder99

Reputation: 1

We chaged the IP tables at Host machine which effected the docker networking so simply restart the docker service resolved my problem.

Upvotes: 0

stelios.anastasakis
stelios.anastasakis

Reputation: 1196

neither of those worked for me, running on ubuntu with current latest versions of prometheus and grafana. what did the trick was to run grafana with this host: sudo docker run --add-host=host.docker.internal:host-gateway -p 3000:3000 ${imagename} and then use "http://host.docker.internal:9090" as target

Upvotes: 2

Naor Bar
Naor Bar

Reputation: 2209

Looks like you're using the same host for Prometheus and Grafana docker containers;
In this case, setting up http://host.docker.internal:9090 as the Prometheus datasource will do the trick:

enter image description here

Upvotes: 7

MyGodItsFull0fStars
MyGodItsFull0fStars

Reputation: 138

You can simply replace localhost with prometheus in the URL field of Grafana.

Tested it on my device just now and using localhost I received the same error as you did. When I replaced the URL with http://prometheus:9090 and clicked on save it did work as intended.

Upvotes: 12

vahid sabet
vahid sabet

Reputation: 583

Set net flag to use host network and access Prometheus using localhost or 127.0.0.1:

docker run -d --network host --name=prometheus ...

Upvotes: 0

konglee28
konglee28

Reputation: 51

In above, prometheus is the domain name of the prometheus docker, which can be resolved by all dockers within same network.

Upvotes: 2

Vel_More
Vel_More

Reputation: 343

You can use the docker inspect command to find the IP address of the Prometheus container and then replace the localhost word with it.

Upvotes: 15

Nouman Ahmed
Nouman Ahmed

Reputation: 110

I'll suggest you to use docker-compose, which better supports in DNS resolving and your issues of localhost will get resolved.

Upvotes: -1

Related Questions