gcloud app deploy fails: permission denied for [ID] from request [URL]

Question

I'm trying to deploy my React app to Google Cloud App Engine using gcloud app deploy from my computer. This worked fine last weekend, and before.

I get the following output:

$ gcloud app deploy .
Services to deploy:

descriptor:      [REDACTED]
source:          [REDACTED]
target project:  [REDACTED]
target service:  [REDACTED]
target version:  [REDACTED]
target url:      [REDACTED]


Do you want to continue (Y/n)?

Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 0 files to Google Cloud Storage                ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 53d56399-b9ac-4471-8ecb-e06e557523fa status: FAILURE.
Build error details: Build error details not available..
Check the build log for errors: https://console.cloud.google.com/gcr/builds/53d56399-b9ac-4471-8ecb-e06e557523fa?project=995586812495

Following the link in the build log takes me to a Cloud Build log in Google Cloud Console. Below is the last section of that log. There is way more log above this point, but it seems to have gone ok.

Step #1 - "builder": INFO rm_node_modules took 0 seconds
Step #1 - "builder": INFO starting: checking_cached_packages_json_layer
Step #1 - "builder": DEBUG Checking cache for cache_key 75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d
Step #1 - "builder": INFO checking_cached_packages_json_layer took 0 seconds
Step #1 - "builder": INFO build process for FTL image took 0 seconds
Step #1 - "builder": INFO full build took 0 seconds
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": "__main__", fname, loader, pkg_name)
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder": exec code in run_globals
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__.py", line 65, in <module>
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__.py", line 54, in main
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/node/builder.py", line 81, in Build
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/node/layer_builder.py", line 60, in BuildLayer
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 113, in Get
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 137, in _getEntry
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 152, in _getLocalEntry
Step #1 - "builder": File "/ftl-v0.9.0.par/__main__/ftl/common/cache.py", line 175, in getEntryFromCreds
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_image_.py", line 279, in exists
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_image_.py", line 293, in manifest
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_image_.py", line 250, in _content
Step #1 - "builder": File "/ftl-v0.9.0.par/containerregistry/client/v2_2/docker_http_.py", line 364, in Request
Step #1 - "builder": containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '403', 'content-length': '288', 'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked', 'server': 'Docker Registry', '-content-encoding': 'gzip', 'docker-distribution-api-version': 'registry/2.0', 'cache-control': 'private', 'date': 'Sun, 11 Nov 2018 12:13:02 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json'}
Step #1 - "builder": Permission denied for "75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d" from request "/v2/apparat-web/app-engine-tmp/build-cache/ttl-7d/node-cache/manifests/75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d". : None
Finished Step #1 - "builder"
ERROR
ERROR: build step 1 "gcr.io/gae-runtimes/nodejs8_app_builder:nodejs8_8_12_0_20180928_RC00" failed: exit status 1

As you can see, the error message is

Permission denied for "75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d" from request "/v2/apparat-web/app-engine-tmp/build-cache/ttl-7d/node-cache/manifests/75e33f825bb6b991a429a251df57d5c82f04475bb87972004100213e6e245c4d". : None
Finished Step #1 - "builder"

I don't understand what it's telling me is wrong. I am not able to map the 75e33-ID to a user account, and even if I am, what is the resource it's claiming is denying permissions and what can I do about it? It seems like google cloud internals to me and I have no clue how to proceed.

Any ideas would be much appreciated.

Things I've tried

• gcloud auth revoke and gcloud auth login to ensure I'm logged in with the correct account for the current project
• gcloud config list and verified the current project is correct.
• When I run gcloud app deploy I've also verified that the project information is correct when it prompts for confirmation before performing the deploy
• Verified my user account is (still) added with correct Roles in Google Cloud Console IAM

Answer 1

I had a similar error message, and turned out to be billing related.

If you are going from free credits to credit card-based billing, you need to manually go in and 'activate' that paid billing, even if your CC is already on file.

Answer 2

I had a very similar problem (identical, except that I was on python37 standard). The logs had this in it:

Step #1 - "builder": Permission denied for "d7dab0ac46167635bcee11cda0f4dbd2963e1cfeaf021404fa295f99c37070e3" from request "/v2/<project-id>/app-engine-tmp/build-cache/ttl-7d/python-cache/manifests/d7dab0ac46167635bcee11cda0f4dbd2963e1cfeaf021404fa295f99c37070e3". : None

It's actually a GAE configuration problem. In my case it was that I deleted the database (moved it to a different service) and disabled billing.

Shockingly, I solved it by simply re-enabling billing for the project...

Answer 3

You've got the same error as me ... in your traces :

Step #1 - "builder": Traceback (most recent call last): Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main Step #1 - "builder": "__main__", fname, loader, pkg_name) Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code

Like me, you have an older version (py27) which seems to break the full chain I'm searchine too Can't upload a new py37 version ;-(