CODEWITHSUNDEEP
google-container-registry
jamiet
jamiet

Reputation: 12264

Permission failure when pulling from gcr.io

I have 2 VMs running on Google Compute Engine. They are identical except for the fact that they are running under different service accounts.
Both of those service accounts have (as far as I can tell) identical permissions on the buckets used by gcr.io enter image description here
enter image description here

The init script that runs when the VM starts up pulls a docker container from gcr.io, on the VM running as [email protected] the pull succeeds:

Unable to find image 'gcr.io/project-id/gdp/jupyterlab-py2-spark-notebook:1.9' locally
1.9: Pulling from project-id/gdp/jupyterlab-py2-spark-notebook
bc51dd8edc1b: Pulling fs layer
b56e3f6802e3: Pulling fs layer

on the VM running as [email protected] the pull fails:

Unable to find image 'gcr.io/project-id/gdp/jupyterlab-py2-spark-notebook:1.9' locally
/usr/bin/docker: Error response from daemon: pull access denied for gcr.io/project-id/gdp/jupyterlab-py2-spark-notebook, repository does not exist or may require 'docker login': denied: Permission denied for "1.9" from request "/v2/project-id/gdp/jupyterlab-py2-spark-notebook/manifests/1.9"

I was under the impression that having identical permissions on the bucket should be sufficient hence I'm wondering what other permissions are required in order to make this work. Could anyone suggest something?

UPDATE. I used toolbox (https://cloud.google.com/container-optimized-os/docs/how-to/toolbox) to verify that the permissions on the bucket are not the same for those two accounts:

# gsutil ls gs://artifacts.project-id.appspot.com
gs://artifacts.project-id.appspot.com/containers/

# gsutil ls gs://artifacts.project-id.appspot.com
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to artifacts.project-id.appspot.com.

Clearly that's the cause of the issue, though I find it very strange that my screenshots above from the GCP Console suggest different. I am continuing to investigate.

Upvotes: 0

Views: 1105

Answers (1)

jamiet
jamiet

Reputation: 12264

This turned out to be a problem that is all too familiar to us because we are constantly creating infrastructure, tearing it down, and standing it up again. When doing so, particularly when those operations don’t occur cleanly (as was the case today), we can find ourselves in a position whereby roles are assigned to an old instance of a service account. The console will tell you that the account has roles assigned to it but that’s actually not the case. We encounter this problem often.

The solution on this occasion was to tear down all the infrastructure cleanly then recreate it again, including the service account that was exhibiting the problem.

Upvotes: 1

Related Questions