CODEWITHSUNDEEP
docker
x300n
x300n

Reputation: 331

How to troubleshoot Docker permission issues with Prometheus/Grafana/Jenkins?

I run into permission issues while trying to run containers from these images:

docker run -d -p 9090:9090 prom/prometheus
docker run -d -p 3000:3000 grafana/grafana 
docker run -d -p 49001:8080 -v jenkins-data:/var/jenkins_home jenkins

And they all result into permission denied error although user is added to docker group and am able to run docker without sudo. The only way around is to run the container as --user root:

docker run -d -p 9090:9090 --user root prom/prometheus

I examined prometheus and by looking into /etc/passwd I find user 'nobody' which I suppose the container is meant to run as, but I still get permission denied unless --user root:

root:x:0:0:root:/root:/bin/sh
....
nobody:x:65534:65534:nobody:/home:/bin/false

I thought containers shouldn't be run as root.

EDIT

Prometheus:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "chdir to cwd ("/prometheus") set in config.json failed: permission denied": unknown.

Grafana:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "chdir to cwd ("/usr/share/grafana") set in config.json failed: permission denied": unknown.

Jenkins:

/usr/local/bin/jenkins.sh: line 5: /var/jenkins_home/copy_reference_file.log: Permission denied

Upvotes: 0

Views: 2407

Answers (2)

snakecharmerb
snakecharmerb

Reputation: 55913

I got the same error trying to build a Dockerfile on Debian Bullseye:

OCI runtime create failed: container_linux.go:367: starting container process caused: chdir to cwd ("/some/dir/path") set in config.json failed: permission denied: unknown

The cause was that after WORKDIR had been set to /some/dir/path there was a directive to switch to the postgres user:

USER postgres

but the postgres user did not have permissions for /some/dir/path, hence the error. The root cause seems to be the the version of runc distributed with Debian

$ runc --version
runc version 1.0.0~rc93+ds1
commit: 1.0.0~rc93+ds1-5+deb11u2
spec: 1.0.2-dev
go: go1.15.15
libseccomp: 2.5.1

executes chdir with the container user rather than the runc process user. The details are explored in great detail in this blog post (not written by me). The problem does not seem to occur with later versions of runc; as Debian does not currently provide a later version I worked around by changing WORKDIR to the postgres user's home directory and changing it back to its previous value once I had finished using the postgres user*.

* This seems a rather crude approach, but it worked.

Upvotes: 0

x300n
x300n

Reputation: 331

User namespaces were not enabled on my system and I needed to configure and restart the daemon by modifying /etc/docker/daemon.json and adding:

"userns-remap": "username" (username is my current user on the docker host)

To check if userns was enabled on the system I used docker info under Security Options

For more info:

Upvotes: 1

Related Questions