gcloud docker push results in "denied: Token exchange failed for project 'gcp-project-id-example'."

Question

I have run gcloud auth login, gcloud auth configure-docker, gcloud components install docker-credential-gcr, gcloud config set project gcp-project-id-example.

I have pushed to this repository previously so I'm a bit surprised it isn't working now? I've authenticated with gcloud auth login and my user has full editor permissions.

sudo gcloud docker -- push eu.gcr.io/gcp-project-id-example/pipelinebuild:latest
WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03.

As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.

See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker

The push refers to repository [eu.gcr.io/gcp-project-id-example/pipelinebuild]
09a1efc2708d: Preparing
484c62332bc0: Preparing
737446294222: Preparing
5330921097a0: Preparing
898d09fcad4e: Preparing
8ebe2c7c93e3: Waiting
2d360789868b: Waiting
43c7850e5ceb: Waiting
212ad79ba733: Waiting
b12c0a65bf50: Waiting
2ec89235b54b: Waiting
770a49082d40: Waiting
2a4ee56ebd9d: Waiting
f38582ca1d15: Waiting
3cc68fcb53a4: Waiting
577d10d964a3: Waiting
96e5efb05969: Waiting
aae94198c5bb: Waiting
9e5b0f110abc: Waiting
bddf843523ce: Waiting
6ab9447934c9: Waiting
9cc1209e0dce: Waiting
072f13fb321e: Waiting
0926f7bf84b3: Waiting
cdb414de0edf: Waiting
eceffb9b1d52: Waiting
6219baf3e782: Waiting
c9189dccc6a7: Waiting
93715b5af77e: Waiting
032237575276: Waiting
5f70bf18a086: Waiting
0d81735d8272: Waiting
982549bd6b32: Waiting
8698b31c92d5: Waiting
denied: Token exchange failed for project 'gcp-project-id-example'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

---

I've also tried:

docker login -u _json_key --password-stdin https://eu.gcr.io < aysc.json:

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

But still get the same behavior with the gcloud docker -- push ... or just the docker push ....

---

I've just created a brand new GCP project from scratch and I'm still getting the same behavior. I think something must be reconfigured on my system.

Why am I getting this error?

Answer 1

I hit the similar issue while i was trying to upload the docker image to GCR from container optimized OS, i ran the following sequence of command,

1. Created a service account and assigned Storage Admin privileges.
2. Downloaded the JSON key
3. Executed docker-credential-gcr configure-docker
4. Logged in with docker command - docker login -u _json_key -p "$(cat ./mygcrserviceaccount.JSON)" https://gcr.io
5. Tried pushing the image gcr - docker push gcr.io/project-id/imagename:tage01

It failed with following error,

denied: Token exchange failed for project 'project-id'. Caller does not have permission 'storage.buckets.create'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

I tried giving every possible permission to my service account through IAM role but it would fail with same error.

After reading this issue i did the following changes,

1. Removed the docker config directory rm -rf ~/.docker
2. Executed docker-credential-gcr configure-docker
3. Stored the JSON key into variable named GOOGLE_APPLICATION_CREDENTIALS GOOGLE_APPLICATION_CREDENTIALS=/path/to/mygcrserviceaccount.JSON

4. Logged in with docker command - docker login -u _json_key -p "$(cat ${GOOGLE_APPLICATION_CREDENTIALS})" https://gcr.io

5. Executed docker push command - docker push gcr.io/project-id/imagename:tage01

Voila, it worked like a charm!

Answer 2

Run this command to activate token again -

gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io

courtesy

Answer 3

When you run gcloud auth configure-docker as a normal user it saves a configuration file in ~/.docker/config.json - and this file is not used afterwards when running via the root user or sudo...

So an easy work around is sudo -u $USER docker push gcr.io/example/example:latest

Answer 4

After creating and downloading the keyfile.json I export GOOGLE_APPLICATION_CREDENTIALS to the location of the file

export GOOGLE_APPLICATION_CREDENTIALS=/var/lib/jenkins/gcp/keyfile.json

this is what I did that worked for me:

docker login -u _json_key -p "`cat ${GOOGLE_APPLICATION_CREDENTIALS}`" https://gcr.io

hope it helps

Answer 5

Seems there are some issues with using sudo (which I need to access docker on my machine).

Worked fine when run directly from root... So probably a bug.

I think it's caused by the cache of previous service account with the same name (see here https://github.com/kubernetes/kubernetes/issues/34395).

Answer 6

Before running docker login, try:

gcloud auth activate-service-account --key-file new-service-account-credentials.json 

This has to be done only once.

Answer 7

I doubt the issue is with sudo. I was able to do "docker push" to gcr.io/some-registry/some-image without using sudo. Then suddenly, I wasn't able to run the same "docker push" command and got the "denied: Token exchange failed for my-gcp-project.