Issues while deploying to google cloud app-engine

Question

I am trying to deploy my springboot application to google cloud app engine flex environment. The deployment is done using docker image. Follwoing is the error I see:

Error Response: [13] Flex operation projects/<project_id>/regions/us-central1/operations/a951c052-5f1a-4c5a-b8b9-45cd7a8ab899 error [INTERNAL]: An internal error occurred while processing task /app-engine-flex/update_flex_deployment/flex_update_instances>2024-03-19T19:49:16.781Z58234.jc.2

My dockerfile is:

FROM gcr.io/google-appengine/openjdk:8-2022-09-09-195526
COPY ./app.jar $APP_DESTINATION
ENTRYPOINT [exec java -showversion -agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/var/log/app_engine,--alsologtostderr=true -Xms3276M -Xmx3276M -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:+PrintCommandLineFlags -jar app.jar " ]

My app.yaml file is :

runtime: custom
env: flex

runtime_config:
  jdk: openjdk8

#service: api

handlers:
- url: /.*
  script: this field is required, but ignored
  secure: always

automatic_scaling:
  min_num_instances: 2
  max_num_instances: 3
  cool_down_period_sec: 120
  cpu_utilization:
    target_utilization: 0.6

service: service-agent

network:
  name: <network_name>
  subnetwork_name: <subnet_name>

readiness_check:
  path: '/check/ready'
  check_interval_sec: 30
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2
liveness_check:
  path: '/check/live'
  check_interval_sec: 30
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2

resources:
  cpu: 4
  memory_gb: 4

Not sure why I am getting this error. I was able to do a deploy on 12-21-2023 with exactly same config and code base. The logs from google only give generic message "An internal error occurred while processing" which is not very helpful. I used to verbosity as debug with command:

gcloud app deploy --no-promote --verbosity=debug

Here is more detailed log I get with verbosity set to debug :

Updating service [service-agent] (this may take several minutes)...⠶DEBUG: Operation [apps/<project_id>/operations/d18182ba-465e-4a53-9210-69890ffb2387] complete. Result: {                                                                                                  
    "done": true,
    "error": {
        "code": 13,
        "message": "Flex operation projects/<project_id>/regions/us-central1/operations/7c540852-3285-4670-8b36-50b845920a6d error [INTERNAL]: An internal error occurred while processing task /app-engine-flex/update_flex_deployment/flex_update_instances>2024-03-19T21:15:00.522Z48869.jf.0: Count of updating instances: none: 1\ncreating: 1\nrecreating: 0\ndeleting: 0\nabandoning: 0\nrestarting: 0\nrefreshing: 0\ncreating_without_retries: 0\nverifying: 0\n"
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "createVersionMetadata": {
            "cloudBuildId": "a155b963-460a-4eee-b3d4-a812bd17e211"
        },
        "insertTime": "2024-03-19T21:13:06.323Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/<project_id>/services/service-agent/versions/20240319t151301",
        "user": "<email_id>"
    },
    "name": "apps/<project_id>/operations/d18182ba-465e-4a53-9210-69890ffb2387"
}
Updating service [service-agent] (this may take several minutes)...failed.    


DEBUG: (gcloud.app.deploy) Error Response: [13] Flex operation projects/<project_id>/regions/us-central1/operations/7c540852-3285-4670-8b36-50b845920a6d error [INTERNAL]: An internal error occurred while processing task /app-engine-flex/update_flex_deployment/flex_update_instances>2024-03-19T21:15:00.522Z48869.jf.0: Count of updating instances: none: 1
creating: 1
recreating: 0
deleting: 0
abandoning: 0
restarting: 0
refreshing: 0
creating_without_retries: 0
verifying: 0
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
    resources = calliope_command.Run(cli=self, args=args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 815, in Run
    resources = command_instance.Run(args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/surface/app/deploy.py", line 120, in Run
    return deploy_util.RunDeploy(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 677, in RunDeploy
    deployer.Deploy(
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 471, in Deploy
    self.api_client.DeployService(new_version.service, new_version.id,
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 247, in DeployService
    return operations_util.WaitForOperation(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 307, in WaitForOperation
    completed_operation = waiter.WaitFor(
                          ^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 261, in WaitFor
    operation = PollUntilDone(
                ^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 322, in PollUntilDone
    operation = retryer.RetryOnResult(
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 249, in RetryOnResult
    if not should_retry(result, state):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 182, in IsDone
    raise OperationError(requests.ExtractErrorMessage(
(requests.ExtractErrorMessage(
googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [13] Flex operation projects/oit-cbe-2-0-prod/regions/us-central1/operations/e3e8f436-df2f-413a-a34d-b092ecadbf0d error [INTERNAL]: An internal error occurred while processing task /app-engine-flex/update_flex_deployment/flex_update_instances>2024-03-19T22:14:03.647Z44215.jc.0: Timed out: GraphFuture{key=@com.google.apps.framework.producers.PrivateVisibility(annotation="@com.google.apphosting.flex.lib.compute.ApplyUpdatesToAllInstancesProducerModule$AreAllInstancesUpdated", module=com.google.apphosting.flex.lib.compute.ApplyUpdatesToAllInstancesProducerModule.class) java.lang.Void, delegate=AsyncCatchingFuture@76c63eea[status=PENDING, info=[inputFuture=[com.google.common.labs.concurrent.RetryingFuture@7801de42[status=PENDING, info=[futureSupplier=[com.google.apps.framework.producers.RetryRunner$1@47633a08], shouldContinue=[com.google.apps.framework.producers.RetryRunner$EqualsClassPredicate@225e1a35], strategy=[uniformDelay{delay=10000, tries=60}], tries=[59], activeTry=[SettableFuture@7c3e559a[status=PENDING, setFuture=[AsyncCatchingFuture@393cd479[status=PENDING, info=[inputFuture=[AsyncTransformFuture@7eca8cda[status=PENDING, info=[inputFuture=[AsyncTransformFuture@84fae8f[status=PENDING, setFuture=[TrustedListenableFutureTask@653a899[status=PENDING, info=[task=[running=[NOT STARTED YET], com.google.common.util.concurrent.Callables$$Lambda$3483/0x0000000802a81c40@59132b36]]]]]], function=[com.google.common.labs.concurrent.RetryingFuture$$Lambda$612/0x000000080179b440@dd9f44e]]]], exceptionType=[class java.lang.Exception], fallback=[com.google.common.labs.concurrent.RetryingFuture$$Lambda$613/0x00000008018a8040@5640e919]]]]]]]]], exceptionType=[class java.lang.Throwable], fallback=[com.google.common.util.concurrent.Futures$1@513a83e8]]]}


googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [13] Flex operation projects/<project_id>/regions/us-central1/operations/7c540852-3285-4670-8b36-50b845920a6d error [INTERNAL]: An internal error occurred while processing task /app-engine-flex/update_flex_deployment/flex_update_instances>2024-03-19T21:15:00.522Z48869.jf.0: Count of updating instances: none: 1
creating: 1
recreating: 0
deleting: 0
abandoning: 0
restarting: 0
refreshing: 0
creating_without_retries: 0
verifying: 0

Also, the issue does not seem to be related to java-8 support because I was able to deploy the same code base and the same config as above on different project with similar project settings.

Answer 1

The error [INTERNAL]: An internal error occurred while processing task /app-engine-flex/update\_flex\_deployment/flex\_update\_instances is not documented in GCP's documentation - Troubleshoot App Engine errors. As such, you might need to contact Google Cloud Support to check if there is an issue in the backend.

As a workaround, you can try to deploy and promote a new version of your service for example by running this command:

gcloud app deploy app.yaml \
  --image-url=gcr.io/<project_id>/openjdk:8-2022-09-09-195526 \
  --quiet --version v1.1 --promote

This will create a new set of instances (new MIG) with the new boot disk.