Deploying Golang AppEngine on Google Cloud Build spawns infinite builds

Question

I'm trying to set up a Cloud Build pipeline to deploy a golang app to App Engine, but it appears to spawn infinite versions of itself and then times out.

To start with, here's my configuration.

steps:
- name: "gcr.io/cloud-builders/go"
  args:
    - get
    - "-u"
    - "-d"
    - "github.com/didip/tollbooth"
    - "github.com/lib/pq"
    - "github.com/stretchr/testify"
    - "github.com/go-redis/redis"
    - "cloud.google.com/go/pubsub"
  dir: "/workspace"
  volumes:
    - name: 'go'
      path: '/gopath'
  env:
    - "GOPATH=/gopath"
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy", "--stop-previous-version"]
  dir: "/workspace"
  volumes:
    - name: 'go'
      path: '/gopath'
  env:
    - "GOPATH=/gopath"

Here's the relevant section of app.yaml:

service: "myservice"
runtime: custom
env: flex

When I deploy to the branch I've configured it to watch, it starts the build just fine, and finishes step 0. Then it starts step 1, which is gcloud app deploy, which appears to recursively spawn yet another instance of the pipeline (see logs below). This continues until everything times out.

Here's the beginning of the log (towards the end you'll see that it has recursively started step 0 inside step 1 again).

starting build "e5ad47ed-5332-4bc7-b4ac-618e243f05a5"

FETCHSOURCE
Fetching storage object: gs://staging.myproject-event.appspot.com/asia.gcr.io/myproject-event/appengine/myservice.20190130t095428:latest#1548842070817530
Copying gs://staging.myproject-event.appspot.com/asia.gcr.io/myproject-event/appengine/myservice.20190130t095428:latest#1548842070817530...
/ [0 files][ 0.0 B/ 10.9 KiB] 
-
- [1 files][ 10.9 KiB/ 10.9 KiB] 
Operation completed over 1 objects/10.9 KiB. 
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/go
Step #0: Documentation at https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/go/README.md
Step #0: Running: go get -u -d github.com/didip/tollbooth github.com/lib/pq github.com/stretchr/testify github.com/go-redis/redis cloud.google.com/go/pubsub
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #1: Services to deploy:
Step #1: 
Step #1: descriptor: [/workspace/app.yaml]
Step #1: source: [/workspace]
Step #1: target project: [myproject-event]
Step #1: target service: [myservice]
Step #1: target version: [20190130t095546]
Step #1: target url: [https://myservice-dot-myproject-event.appspot.com]
Step #1: 
Step #1: 
Step #1: Do you want to continue (Y/n)? 
Step #1: WARNING: Unable to verify that the Appengine Flexible API is enabled for project [myproject-event]. You may not have permission to list enabled services on this project. If it is not enabled, this may cause problems in running your deployment. Please ask the project owner to ensure that the Appengine Flexible API has been enabled and that this account has permission to list enabled APIs.
Step #1: Beginning deployment of service [myservice]...
Step #1: Building and pushing image for service [myservice]
Step #1: Started cloud build [b41069a6-2ef7-4eaf-8b49-de36bc620be2].
Step #1: To see logs in the Cloud Console: https://console.cloud.google.com/gcr/builds/b41069a6-2ef7-4eaf-8b49-de36bc620be2?project=954549095871
Step #1: ----------------------------- REMOTE BUILD OUTPUT ------------------------------
Step #1: starting build "b41069a6-2ef7-4eaf-8b49-de36bc620be2"
Step #1: 
Step #1: FETCHSOURCE
Step #1: Fetching storage object: gs://staging.myproject-event.appspot.com/asia.gcr.io/myproject-event/appengine/myservice.20190130t095546:latest#1548842148386512
Step #1: Copying gs://staging.myproject-event.appspot.com/asia.gcr.io/myproject-event/appengine/myservice.20190130t095546:latest#1548842148386512...
Step #1: / [0 files][ 0.0 B/ 10.9 KiB] 
/ [0 files][ 10.9 KiB/ 10.9 KiB] 
-
- [1 files][ 10.9 KiB/ 10.9 KiB] 
Step #1: Operation completed over 1 objects/10.9 KiB. 
Step #1: BUILD
Step #1: Starting Step #0
Step #1: Step #0: Already have image (with digest): gcr.io/cloud-builders/go
Step #1: Step #0: Documentation at https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/go/README.md
Step #1: Step #0: Running: go get -u -d github.com/didip/tollbooth github.com/lib/pq github.com/stretchr/testify github.com/go-redis/redis cloud.google.com/go/pubsub
Step #1: Finished Step #0
Step #1: Starting Step #1
Step #1: Step #1: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #1: Step #1: Services to deploy:

This goes on until I manually kill the last build:

Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: Copying gs://staging.solution360-event.appspot.com/asia.gcr.io/solution360-event/appengine/authproxy.20190130t100329:latest#1548842611410114...
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: / [0 files][ 0.0 B/ 10.9 KiB] 
-
- [1 files][ 10.9 KiB/ 10.9 KiB] 
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: Operation completed over 1 objects/10.9 KiB. 
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: BUILD
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: Starting Step #0
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: Step #0: Already have image (with digest): gcr.io/cloud-builders/go
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: Step #0: Documentation at https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/go/README.md
Step #1: Step #1: Step #1: Step #1: Step #1: Step #1: Step #0: Running: go get -u -d github.com/didip/tollbooth github.com/lib/pq github.com/stretchr/testify github.com/go-redis/redis cloud.google.com/go/pubsub

What I've tried:

• Removing the "--stop-previous-versions" flag from the command.
• Mounting gopath in a different volume, in case that was causing gcloud to recursively descend into it.
• Running just step 1, same result.

See attached screenshot of the actual builds spawning:

Answer 1

So this turned out to be an issue with how Google Cloud Build handles an encounter with the configuration file cloudbuild.yaml. It has to spawn a new build every time it sees one, regardless of whether it has already been seen.

In my case, since the project was tiny, the cloudbuild.yaml and my go source were both at the root of the project. This lead to the following situation:

• The initial git push triggers a build, which reads cloudbuild.yaml.
• The build sees that there's a cloudbuild.yaml in the current working directory and spawns a new build for that.

I suppose this is useful as you can have multiple build configs for various things in your projects, that will be built as and when encountered.

Moving the go source to src/ (away from the cloudbuild) fixed the issue.

tl;dr DON'T PUT YOUR cloudbuild.yaml IN THE WORKING DIRECTORY.

Answer 2

You do NOT have permissions to deploy app and enable Appenginge Flexible API.

In the log, there is a warning:

WARNING: Unable to verify that the Appengine Flexible API 
is enabled for project [project-name]. 
You may not have permission to list enabled services on this project.
If it is not enabled, this may cause problems in running your deployment.
Please ask the project owner to ensure that the Appengine Flexible API
has been enabled and that this account has permission to list enabled APIs.

Steps to streamline deployment of the AppEngine Flexible Apps:

1. As a project owner: enable Appengine Flexible API + deploy app for the first time, and:
2. Grant permissions to deploy AppEngine app for the user that you are using to perform that tasks.

Answer 3

i haven't been able to recreate the infinite builds but a couple things

• custom runtimes require a Dockerfile, have you tried running your service in the go runtime?

  • https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build

  • https://cloud.google.com/appengine/docs/go/

  • https://cloud.google.com/appengine/docs/flexible/go/quickstart

• is there a specific reason you are using volumes instead of just setting the project root in the env?

  • https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/go/examples/hello_world/cloudbuild.yaml
  • https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/go/examples