GAE deploying results in an error

Question

I'm trying to deploy my projet on Google App Engine. It previously worked successfully but since yesterday, I can't deploy it.

I get the following error :

File upload done.
Updating service [default]...failed.                                            
ERROR: (gcloud.app.deploy) Error Response: [13] The system encountered a fatal error

When looking at the log file, I get the following :

2017-02-17 10:15:51,710 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:15:56,876 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:02,173 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:07,352 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:12,654 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:17,817 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:23,119 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:28,296 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:33,475 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:38,650 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.
2017-02-17 10:16:43,838 DEBUG    root            Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] complete. Result: {
    "metadata": {
        "target": "apps/my-project/services/default/versions/20170217t100612", 
        "method": "google.appengine.v1.Versions.CreateVersion", 
        "user": "user@gmail.com", 
        "insertTime": "2017-02-17T09:06:34.800Z", 
        "ephemeralMessage": "Creating new version configuration in service \"20170217t100612\".", 
        "endTime": "2017-02-17T09:16:43.099Z", 
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1"
    }, 
    "done": true, 
    "name": "apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae", 
    "error": {
        "message": "The system encountered a fatal error", 
        "code": 13
    }
}
2017-02-17 10:16:43,838 DEBUG    root            (gcloud.app.deploy) Error Response: [13] The system encountered a fatal error
Traceback (most recent call last):
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 742, in Execute
    resources = args.calliope_command.Run(cli=self, args=args)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 739, in Run
    resources = command_instance.Run(args)
  File "/opt/google-cloud-sdk/lib/surface/app/deploy.py", line 59, in Run
    return deploy_util.RunDeploy(args)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 413, in RunDeploy
    all_services)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 255, in Deploy
    endpoints_info)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 141, in DeployService
    operation)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 142, in WaitForOperation
    encoding.MessageToPyValue(completed_operation.error)))
OperationError: Error Response: [13] The system encountered a fatal error
2017-02-17 10:16:43,841 ERROR    root            (gcloud.app.deploy) Error Response: [13] The system encountered a fatal error
2017-02-17 10:16:44,726 DEBUG    root            Metrics reporting process started...

There is a lot of similar line like "2017-02-17 10:15:51,710 DEBUG root Operation [apps/my-project/operations/4aced3ef-a018-4913-9a5b-b3f8f13ed4ae] not complete. Waiting 5s.".

When I take a look at the "Activity" tab on the GAE web console, here's what I see :

Sorry it's in French. It's written :

10:16 - Successful operation : create a version of AppEngine
10:06 - Create a version of AppEngine

So I suspect the deployment worked, but took a lot of time to be deployed (10 minutes). Also you can see that it always take 10 minutes to be successfully done, on all my attempts.

What is wrong?

Answer 1

From recent google-cloud-sdk issue 1443 (same symptoms as far as I can tell):

We suspect from the backend logs that the application is actually deploying successfully but that part that checks if the application is ready to serve is in fact returning a false negative and thus, returning the error message you received. It is then stuck in this inaccurate state. I've added the internal tracking number of that issue to this report as well.

Can you try issuing requests to that application/service before, during and after attempting to deploy? The hypothesis is that this request should provide a positive response to a health check. If this is received during the troubling deployment, it may trigger the deployment to returning a positive when checking if the version is ready to serve.

And the workaround suggestion worked:

As per your request: I issued requests before, during and after deployment. The deployment was successful. 'gcloud app deploy' returned normally without any errors reported I tested the GAE app and indeed the code running in the GAE project is the newly deployed version.