CODEWITHSUNDEEP
dockerdocker-composeprometheus
Nyxynyx
Nyxynyx

Reputation: 63599

Prometheus (in Docker container) Cannot Scrape Target on Host

Prometheus running inside a docker container (version 18.09.2, build 6247962, docker-compose.xml below) and the scrape target is on localhost:8000 which is created by a Python 3 script.

Error obtained for the failed scrape target (localhost:9090/targets) is

Get http://127.0.0.1:8000/metrics: dial tcp 127.0.0.1:8000: getsockopt: connection refused

Question: Why is Prometheus in the docker container unable to scrape the target which is running on the host computer (Mac OS X)? How can we get Prometheus running in docker container able to scrape the target running on the host?

Failed attempt: Tried replacing in docker-compose.yml

networks: 
  - back-tier
  - front-tier

with

network_mode: "host"

but then we are unable to access the Prometheus admin page at localhost:9090.

Unable to find solution from similar questions

docker-compose.yml

version: '3.3'

networks:
  front-tier:
  back-tier:

services:

  prometheus:
    image: prom/prometheus:v2.1.0
    volumes:
      - ./prometheus/prometheus:/etc/prometheus/
      - ./prometheus/prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    networks:
      - back-tier
    restart: always


  grafana:
    image: grafana/grafana
    user: "104"
    depends_on:
      - prometheus
    ports:
      - 3000:3000
    volumes:
      - ./grafana/grafana_data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    env_file:
      - ./grafana/config.monitoring
    networks:
      - back-tier
      - front-tier
    restart: always

prometheus.yml

global:
scrape_interval:     15s 
evaluation_interval: 15s 

external_labels:
    monitor: 'my-project'

- job_name: 'prometheus'

    scrape_interval: 5s

    static_configs:
        - targets: ['localhost:9090']


- job_name: 'rigs-portal'

    scrape_interval: 5s

    static_configs:
        - targets: ['127.0.0.1:8000']

Output at http://localhost:8000/metrics

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 65.0
python_gc_objects_collected_total{generation="1"} 281.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 37.0
python_gc_collections_total{generation="1"} 3.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="7",patchlevel="3",version="3.7.3"} 1.0
# HELP request_processing_seconds Time spend processing request
# TYPE request_processing_seconds summary
request_processing_seconds_count 2545.0
request_processing_seconds_sum 1290.4869346540017
# TYPE request_processing_seconds_created gauge
request_processing_seconds_created 1.562364777766845e+09
# HELP my_inprorgress_requests CPU Load
# TYPE my_inprorgress_requests gauge
my_inprorgress_requests 65.0

Python3 script

from prometheus_client import start_http_server, Summary, Gauge
import random
import time

# Create a metric to track time spent and requests made
REQUEST_TIME = Summary("request_processing_seconds", 'Time spend processing request')

@REQUEST_TIME.time()
def process_request(t):
    time.sleep(t)

if __name__ == "__main__":
    start_http_server(8000)
    g = Gauge('my_inprorgress_requests', 'CPU Load')
    g.set(65)

    while True:
        process_request(random.random())

Upvotes: 36

Views: 36315

Answers (4)

Yahya mlaouhi
Yahya mlaouhi

Reputation: 51

i have used host.docker.internal to access the service inside the docker-compose file but it does not work i think you need to add

--add-host=host.docker.internal:host-gateway

you can check the response here How to access host port from docker container

bu the easy way for me was using the IP address (if you are using ubuntu use this command "ifconfig") or the server name as target for my Prometheus service in the prometheus.yaml

  • targets: - your-ip-address:8000

Upvotes: 0

rbento
rbento

Reputation: 11598

Below is an example of running Prometheus on Docker for macOS which causes Prometheus to scrape a simple Spring Boot application running on localhost:8080:

Bash

docker run --rm --name prometheus -p 9090:9090 -v /Users/YourName/conf/prometheus.yml:/etc/prometheus/prometheus.yml -d prom/prometheus

/Users/YourName/conf/prometheus.yml

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 

scrape_configs:
  - job_name: 'spring-boot'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:8080']

In this case it is the use of the special domain host.docker.internal instead of localhost that causes the host to be resolved from the container on a macOS as the config file is mapped into the Prometheus container.

Environment

  • Macbook Pro, Apple M1 Pro
  • Docker version 20.10.17, build 100c701
  • Prometheus 2.38

Upvotes: 18

Nader Elshehabi
Nader Elshehabi

Reputation: 139

For reference for people who might find this question through search, this is supported now as of Docker 20.10 and above. See the following link:

How to access host port from docker container

and:

https://github.com/docker/for-linux/issues/264#issuecomment-823528103

Upvotes: 6

Alexandre Juma
Alexandre Juma

Reputation: 3303

While not a very common use case.. you can indeed connect from your container to your host.

From https://docs.docker.com/docker-for-mac/networking/

I want to connect from a container to a service on the host

The host has a changing IP address (or none if you have no network access). From 18.03 onwards our recommendation is to connect to the special DNS name host.docker.internal, which resolves to the internal IP address used by the host. This is for development purpose and will not work in a production environment outside of Docker Desktop for Mac.

Upvotes: 39

Related Questions