CODEWITHSUNDEEP
amazon-web-servicesdockeramazon-s3docker-composeprometheus
nico_1512
nico_1512

Reputation: 130

Prometheus "operation not permitted"

I'm configuring a monitoring machine to get metrics from other machines. I'm hosting Grafana and Prometheus in a AWS EC2 machine (t2.micro) with docker-compose.

For Prometheus I've mounted an S3 bucket with s3fs and passing the mounted volume to Prometheus container. I get this row by running df -h | grep s3fs:

Filesystem      Size  Used Avail Use% Mounted on
...
s3fs             64P     0   64P   0% /mnt/s3-bucket
...

Here's my compose file:

version: '3.9'

services:
  
  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: unless-stopped
    extra_hosts:
     - "host.docker.internal:host-gateway"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro        
      - ./grafana/data:/var/lib/grafana                   
    user: "0"                                             
    ports:
      - "3000:3000"

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    network_mode: host
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - /mnt/s3-bucket/prometheus:/prometheus
#       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#       HERE IS WHERE I MOUNT MY S3 BUCKET (of course the prometheus folder exists in that path)
      - ./prometheus:/etc/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - "9090:9090"
    expose:
      - 9090
    user: "0"

I run docker compose up -d to create my containers and everything seems working fine. After some time, the Prometheus container dies for no reason. With docker logs prometheus i get this error:

panic: 2 errors: replace file: rename data/chunks_head/000038.tmp data/chunks_head/000038: operation not permitted; unlinkat data/chunks_head/000038.tmp: operation not permitted

I've triple checked the permissions on that folder; here's what I've tried:

sudo chmod 777 /mnt/s3-bucket/prometheus

sudo chmod -R a+rwxX /mnt/s3-bucket/prometheus

sudo chown 1000:1000 /mnt/s3-bucket/prometheus # (before the container had as user 1000)

sudo chown prometheus:prometheus /mnt/s3-bucket/prometheus # (before the container had as user prometheus)

Everything I've tried so far resulted with the same error: operation not permitted

Any suggestion?

Upvotes: 0

Views: 378

Answers (1)

Cloudlady
Cloudlady

Reputation: 763

Have you double checked the AWS permissions of the keys used to mount the s3 bucket using s3fs? Make sure that it has the right permissions like PutObject, ReadObject, etc.

Upvotes: 0

Related Questions