Network between two cloud build builder containers

Question

I am trying to run my Go app's integration test in cloudbuild. Part of this testing requires my app-under-test to write to and read from Redis. My approach to do this was to start up a custom-build container in build STEP-1 that runs redis-server in the foreground using this command for the step args with a bash entrypoint:

docker run -d -e ENV_A=123 -e ENV_b=456 my-custom-redis-image

Using -d ensures it runs detached allowing that build step to complete but leaving Redis up and listening for connections. Redis is configured to listen on 0.0.0.0:6379. Then, in say STEP-2, I want to compile my Go app integration tests and run them as follows using the golang:1.16.4 builder container:

go test -tags=integration ./...

When that test runs, it will create a Redis client that wants to connect to the running Redis instance from STEP-1. However, I do not know what IP to configure in my client that can reach the Redis instance running in the STEP-1. I understand that there is a "cloudbuild" network that is used in cloudbuild for all the docker instances. My confusion is how to configure my client to talk to Redis in another running container. It is even possible?

If not, how do you handle test dependencies like this where tests needs N external services to work? I have unit tests and they work fine because, unit tests do not have any external dependencies. However, my integration test needs to not only use Redis, but also needs to connect to GCS, Pub/Sub, and BQ to complete various test scenarios and these test are much more black box like as compared to unit tests. The connections to those GCP services work fine from the integration test because they are public Google APIs. I just need to figure out how to connect to my test Redis server running in cloud build. These tests need to communicate with the real services and not mocks or fakes - so that is not a option.

Appreciate any guidance on this.

Answer 1

Thanks to @JohnHanley for his pointer to the link. In that link, I came to understand what I needed to do to get my Go integration test to be able to access Redis where Redis was started in one CloudBuild step and my Go integration test was run in a subsequent build step.

For anyone who is interested in what this might look like, below are the two build steps in my CloudBuild YAML as well as the Docker compose file I am using to automate my integration test runs. This demonstrates the general approach whereby you can spin up one or more containers that provide services needed to support your integrations tests.

CloudBuild YAML example file:

substitutions:
  _BITBUCKET_CLONE_PRIVATE_KEY: BITBUCKET_CLONE_PRIVATE_KEY
  _RESOURCES_PROJECT: my-gcp-resources-project-id
  _GOMODCACHE: /workspace/go/pkg/mod
  _SSHKEYFILE: /root/.ssh/keyfile
steps:
  # Clone application repo from the mirrored source repository in GCP
  - id: clone-repo
    name: gcr.io/cloud-builders/gcloud
    args:
      [
        "source",
        "repos",
        "clone",
        "${REPO_NAME}",
        "--project",
        "${_RESOURCES_PROJECT}",
      ]

  # Check out the commit that caused this build to be triggered
  - id: checkout-commit
    name: gcr.io/cloud-builders/git
    entrypoint: /bin/bash
    dir: ${REPO_NAME}
    args:
      - -c
      - |
        git checkout ${COMMIT_SHA}
        git log -n1

  # We need to start up an instance of Redis before running the integration tests as they need
  # to read and write in Redis. We start it in detached mode (-d) so that Redis server starts up
  # in the container but then does not block this build step. This allows the next build step to
  # run the integration tests against this already-running Redis instance. The compose YAML file
  # must define the cloudbuild network in its configuration. This ensure that Redis, running in
  # this container, will expose port 6379 and hostname "redis" on that cloudbuild network. That
  # same network will be made available inside each cloudbuild builder container that is started
  # in each build step. The next build step is going to run integration tests that expect to
  # connect to "redis:6379" on the cloudbuild network.
  - id: start-redis
    name: "docker/compose:1.19.0"
    args:
      [
        "-f",
        "build/docker-compose.yaml",
        "up",
        "-d",
      ]

    # To build the Go application, we need to do some special setup to allow Go to pull
    # dependent modules, that reside in our private Bitbucket repos, into the build where
    # authentication via SSH keys is the only way we can authenticate without user interaction.
    # First we copy the private key, preloaded into Secret Manager, into the root ssh directory
    # where the SSH client will be looking for it. We then construct an SSH config file that
    # points to that identity file, forces SSH to ONLY try to authenticate with SSH keys, and
    # does not prompt for host authenticity, otherwise, it would require user interaction.
    # Additionally, we must update git config to automatically replace URLs that attempt to
    # access Bitbucket via HTTPS with ones that use SSH, instead. This is an important and
    # non-obvious required setting and is needed because "go get" attempts to access repos via
    # HTTPS only and we have to leverage this special feature of git to effectively rewrite
    # those outbound URLs to use SSH, instead. Finally, we point GOMODCACHE to a directly under
    # /workspace and build and run the integration test. Note that we must specifically define
    # that we are using a specific secret in this build step and use double "$$" when
    # referencing that secret. We set GOMODCACHE to a directory under the workspace directory so
    # that the pulled modules will be preserved across multiple build steps and can be reused by
    # subsequent build steps that compile and run unit tests as well as complile the application
    # binary after all tests pass. This reduces the build times for those subsequent build steps
    # as they only need to pull modules not already in the GOMODCACHE directory.
  - id: integration-test
    name: golang:1.16.4
    entrypoint: /bin/bash
    dir: ${REPO_NAME}/integration_tests
    args:
      - -c
      - |
        mkdir -p /root/.ssh/ && \
        echo "$$BITBUCKET_CLONE_PRIVATE_KEY" > $_SSHKEYFILE && \
        chmod 600 $_SSHKEYFILE && \
        echo -e "Host bitbucket.org\n IdentitiesOnly=yes\n IdentityFile=$_SSHKEYFILE\n StrictHostKeyChecking=no" > /root/.ssh/config && \
        git config --global url."git@bitbucket.org:my-org-name".insteadof "https://bitbucket.org/my-org-name" && \
        go env -w GOMODCACHE=$_GOMODCACHE && \
        CGO_ENABLED=0 go test -count=1 -tags=integration
    secretEnv: ["BITBUCKET_CLONE_PRIVATE_KEY"]
    env:
      # Pass the current project in and tell the integration test framework that we are running
      # in cloudbuild so it can behave accordingly. For example, when I run the integration
      # tests from my workstation, QUASAR_DPLY_ENV=devel1, which is the name of my development
      # environment. The test logic picks up on that and will attempt to connect to a
      # locally-running Redis container on localhost:6379 that I am responsible for running
      # before testing. However, when running here in cloudbuild, the test framework expects to
      # connect to redis:6379 because, in cloudbuild, Redis is running from Docker compose and
      # will be listening on the cloudbuild Docker network and reachable via the hostname of
      # "redis", and not localhost.
      - "DATADIRECTOR_INTTEST_GCP_PROJECT=$PROJECT_ID"
      - "QUASAR_DPLY_ENV=cloudbuild"

# We need access to the clear text SSH private key to use during the Go application build so we
# fetch it here from the Secret Manager to make it available in other build steps
availableSecrets:
  secretManager:
    - versionName: projects/${_RESOURCES_PROJECT}/secrets/${_BITBUCKET_CLONE_PRIVATE_KEY}/versions/1
      env: BITBUCKET_CLONE_PRIVATE_KEY
logsBucket: 'gs://${_RESOURCES_PROJECT}-cloudbuild-logs'

Docker compose YAML file

This is a copy of the compose.yaml file used in the "start-redis" build step in the above example cloudbuild.yaml file:

# cloudbuild.yaml file

version: "3"
services:
  redis:
    image: redis:6.2.5-buster
    container_name: redis
    # We must specific this special network to ensure that the Redis service exposes its
    # listening port on the network that all the other builder containers use. Specifically, the
    # integration tests need to connect to this instance of Redis from a separate running
    # container that is part of a build step that follows the step that runs this compose file.
    network_mode: cloudbuild
    # Ensure that we expose the listening port so that other containers, connected to the same
    # cloudbuild network, will be able to reach Redis
    expose:
      - 6379