Share (parameterized) cloudbuild.yaml between multiple GitHub projects?

Question

I've managed to trigger a Google Cloud Build on every commit to GitHub successfully. However, we have many different source code repositories (projects) on GitHub, that all use Maven and Spring Boot, and I would like all of these projects to use the same cloudbuild.yaml (or a shared template). This way we don't need to duplicate the cloudbuild.yaml in all projects (it'll be essentially the same in most projects).

For example, let's just take two different projects on GitHub, A and B. Their cloudbuild.yaml files could look something like this (but much more complex in our actual projects):

Project A:

steps:
  - name: maven:3.8.6-eclipse-temurin-17-alpine
    entrypoint: mvn
    args: [ 'test' ]
  - name: maven:3.8.6-eclipse-temurin-17-alpine
    entrypoint: mvn
    args: [ 'package', '-Dmaven.test.skip=true' ]
  - name: gcr.io/cloud-builders/docker
    args: [ "build", "-t", "europe-west1-docker.pkg.dev/projectname/repo/project-a", "--build-arg=JAR_FILE=target/project-a.jar", "." ]
images: [ "europe-west1-docker.pkg.dev/projectname/repo/project-a" ]

Project B:

steps:
  - name: maven:3.8.6-eclipse-temurin-17-alpine
    entrypoint: mvn
    args: [ 'test' ]
  - name: maven:3.8.6-eclipse-temurin-17-alpine
    entrypoint: mvn
    args: [ 'package', '-Dmaven.test.skip=true' ]
  - name: gcr.io/cloud-builders/docker
    args: [ "build", "-t", "europe-west1-docker.pkg.dev/projectname/repo/project-a", "--build-arg=JAR_FILE=target/project-b.jar", "." ]
images: [ "europe-west1-docker.pkg.dev/projectname/repo/project-b" ]

The only thing that is different is the jar file and image name, the steps are the same. Now imagine having hundreds of such projects, it can become a maintenance nightmare if we need to change or add a build step for each project.

A better approach, in my mind, would be to have a template file that could be shared:

steps:
  - name: maven:3.8.6-eclipse-temurin-17-alpine
    entrypoint: mvn
    args: [ 'test' ]
  - name: maven:3.8.6-eclipse-temurin-17-alpine
    entrypoint: mvn
    args: [ 'package', '-Dmaven.test.skip=true' ]
  - name: gcr.io/cloud-builders/docker
    args: [ "build", "-t", "europe-west1-docker.pkg.dev/projectname/repo/${PROJECT_NAME}", "--build-arg=JAR_FILE=target/${PROJECT_NAME}.jar", "." ]
images: [ "europe-west1-docker.pkg.dev/projectname/repo/${PROJECT_NAME}" ]

It would then be great if such a template file could be uploaded to GCS and then reused in the cloudbuild.yaml file in each project:

Project A:

steps:
  import:
    gcs: gs:/my-build-bucket/cloudbuild-template.yaml
    parameters: 
      PROJECT_NAME: project-a

Project B:

steps:
  import:
    gcs: gs:/my-build-bucket/cloudbuild-template.yaml
    parameters: 
      PROJECT_NAME: project-b

Does such a thing exists for Google Cloud Build? How do you import/reuse build steps in different builds as I described above? What's the recommended way to achieve this?

Answer 1

You have two readily available solutions on Google Cloud Build:

• encapsulate these build procedures into one or more reusable Docker images
• make use of gcloud builds submit

Eventually, whichever you decide, you will end up with something like this in your project build configurations:

 - name: gcr.io/my-project/maven-spring
   args: ['$REPO_NAME', 'project-a']

Some rationale

I don't believe these are the only two viable options you have to approach the problem of "reusable build code", but I think they play nicely with GCB and other related services you might already have provisioned.

With reusable Docker images, the benefit is that the images run in the same execution context as your build configuration, so you can imagine this is akin to the experience you already have with using existing gcr.io/cloud-builders or community images.

With gcloud builds submit, the benefit is that you can run an entirely separate build in it's own execution context, and either block until it completes or run these asynchronously.

The approach with building your custom Docker image that can perform these common tasks with the sources available on /workspace is a bit more straightforward.

In case that asynchronous builds seems like the right option, then I imagine what you can do is publish a Docker image to a shared Artifact/Container Registry, based on gcr.io/cloud-builders/gcloud, which has all the templates contained within, where on execution sources will be available on the automatically mounted /workspace path, and you can add some more nice touches like validating the Docker run arguments.

Answer 2

I contacted Google Cloud support and they told me that this is not currently available. They are aware of the issue and it's something that they're working on (no eta on when it's going to be available).

Their recommendation, in the meantime, is to use Tekton.