CODEWITHSUNDEEP
securitynetwork-programmingprometheusprometheus-pushgateway
RonZhang724
RonZhang724

Reputation: 575

Access Control for the Prometheus Pushgateway

We have a Prometheus Pushgateway running and listening to metrics push from our AWS Lambda function. However, the URL to the Pushgateway is accessible by the public, which might impose some security issues. We were wondering if there is any way we could add a layer of protection to the Pushgateway so that it is not publicly accessible?

I found this Github thread that may answered this question: https://github.com/prometheus/pushgateway/issues/281

It proposed to set up a reverse proxy in front of the pushgateway. However, I am still confused on how that may actually work? We are currently using Kubernetes to deploy the Prometheus.

Upvotes: 5

Views: 5105

Answers (3)

Nigrimmist
Nigrimmist

Reputation: 12386

You are right, you need reverse proxy here. I also faced with the same issue, so you need nginx in front of your prometheus/pushgateway.

First, install nginx using this article (you can start from Step 8 — Securing Prometheus if you already configured prometheus):

My nginx config :

events { }
http {
upstream prometheus {
      server 127.0.0.1:9090;
      keepalive 64;
}

upstream pushgateway {
      server 127.0.0.1:9091;
      keepalive 64;
}

server {
      root /var/www/example;
      listen 0.0.0.0:80;
      server_name __;      
      location / {
            auth_basic "Prometheus server authentication2";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_pass http://prometheus;
      }  
}


server {
      root /var/www/example;
      listen 0.0.0.0:3001;
      server_name __;      
      location / {
            auth_basic "Pushgateway server authentication";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_pass http://pushgateway;
      } 
}
}

my pushgateway.service file :

[Unit]
Description=Pushgateway
Wants=network-online.target
After=network-online.target

[Service]
User=pushgateway
Group=pushgateway
Type=simple
ExecStart=/usr/local/bin/pushgateway --web.listen-address="127.0.0.1:9091" --web.telemetry-path="/metrics"  --persistence.file="/tmp/metric.store"  --persistence.interval=5m --log.level="info" --log.format="logger:stdout?json=true"

[Install]
WantedBy=multi-user.target

It is important to set : --web.listen-address="127.0.0.1:9091", not ":9091" - so it will be exposed only to localhost.

Through the nginx pushgateway will be accessible on port 3001, port 9091 will be not public. Base authentication will be required to have access or push metrics.

About how to test it using Postman you can find here

Upvotes: 2

Sunny
Sunny

Reputation: 56

You can include authentication in your ingress controller by using a TLS secret as an ingress rule. Here's an example that shows how to generate basic auth for your ingress:

https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

Also, don't forget to include the Python handler function in your client to set the auth header as pointed out here:

https://github.com/prometheus/client_python#handlers-for-authentication

Upvotes: 3

gsood
gsood

Reputation: 96

A suggestion here will be to make the URL of the Pushgateway Internal by using an AWS Internal Load Balancer, create an AWS Private Hosted Zone attach your VPC to this zone after this the next step will be to deploy the lambda in the same VPC.

This should solve the security issue.

Upvotes: 1

Related Questions