Unable to Successfully Build Go-Based Docker Image

Question

When attempting to build a GoLang-based Docker Image, the Docker executor runs into the following error:

. . .go: $GIT_REPO@v1.9.11: reading $GIT_REPO/go.mod at revision v0.0.07: unknown revision v0.0.07

at the following RUN instruction from the Dockerfile used:

RUN go build . . .

where GIT_REPO represents the private repo. full path, including owner and name.

The Docker executor encounters this error with go1.13.x and higher; the Docker executor does not encounter this error with go1.12.x.

The vendor dir. contains all required packages. Tags are confirmed to be present.

Proper SSH keys were even added to the private Go common repo. with successful

git clone . . .

commands outside of building Docker images, but still encountering the same error above.

Answer 1

There are two problems to solve here:

1. How to allow Docker to access local SSH keys, safely?

2. How to tell Go not to use public registry to fetch private packages?

---

Short Answers

1. Since Docker v18.09, there's a built-in solution to handle SSH authentication during the build phase (more). It's also much easier and safer, compared to passing build arguments, and eliminates the need for a multi-stage Docker build.

2. Go has a GOPRIVATE environment variable to identify private packages. (more)

---

Long Answer

Step-by-step:

1. Make sure ssh-agent is setup and knows the SSH key

Github has a quick guide on this subject, explaining the process for different operating systems. See Generating a new SSH key and adding it to SSH agent.

2. Enable BuildKit for Docker

Without BuildKit, docker build won't recognize --ssh option.

From the Docker reference:

Easiest way from a fresh install of docker is to set the DOCKER_BUILDKIT=1 environment variable when invoking the docker build command, such as:

$ DOCKER_BUILDKIT=1 docker build .

To enable docker BuildKit by default, set daemon configuration in /etc/docker/daemon.json feature to true and restart the daemon:

{ "features": { "buildkit": true } }

Docker Desktop users can manage daemon configurations via Preferences > Docker Engine.

4. Update Dockerfile

4.1. Make sure Git uses SSH instead of HTTPs

Go tends to fetch public packages via HTTPs. You can adjust this behavior by updating git configurations:

RUN git config --global url.git@github.com:.insteadOf https://github.com/

You should probably do this on your local machine as well.

4.2. Request SSH access where it's required

Each RUN command the needs SSH access should be mounted with type=ssh. For

Example:

RUN --mount=type=ssh git clone ...

4.3. Make sure Go knows your private packages

Update GOPRIVATE variable:

RUN go env -w GOPRIVATE="github.com/your-org/private-repo"

Putting all of it together, in following sample of a Dockerfile:

FROM golang:1.16.3-alpine3.13
RUN apk update
RUN apk add git openssh
RUN mkdir /app
ADD . /app
WORKDIR /app

# You can replace github.com with any other Git host
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

# Make sure git uses SSH to fetch packages, not HTTPs
RUN git config --global url.git@github.com:.insteadOf https://github.com/

# Make Go knows which packages are private.
RUN go env -w GOPRIVATE="github.com/your-org/private-repo"
# GOPRIVATE is a comma separated list glob-patterns.
# You can use a wildcard to match every repo in an organization:
#   e.g.: GOPRIVATE="github.com/your-org/*"

# Mount the build command with type `ssh`.
RUN --mount=type=ssh go get && go build -o main .

CMD ["/app/main"]

6. Build the image with --ssh option:

Having BuildKit enabled by default:

$ docker build --ssh default -t my-app:latest .

Answer 2

EDIT:

Verify your remote repo in bitbucket.org actually has the v0.0.7 tag you're trying to build against.

While a local build may work if the git tag exists locally - a docker build will pull from the remote source and fail with an error like go.mod at revision v0.0.7: unknown revision v0.0.7 - if the tag does not exist remotely.

To push your local tags to the remote repo:

git push --tags

For more granular tag operations see.

---

Docker builds by default can only access public repos. Since you need access to a private repo, you need to include a read-ssh key to the Docker build process (keys should never be checked into the repo!).

It is critically important, however, you do this in a multi-stage build, so you do not include your SSH keys in the final image.

This blog post walks through all the steps. But to include a working example:

To build the docker image:

SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)" \
    docker build -t "myapp:v0.0.1" --build-arg SSH_PRIVATE_KEY .

And the Dockerfile using a bitbucket.org private repo site:

FROM golang:1.14.6 AS build
  
WORKDIR /bld
COPY *.go go.mod go.sum ./

ARG SSH_PRIVATE_KEY
# ***NEVER*** DO THIS IN A SINGLE-STAGE DOCKER BUILD (see below)
RUN \
        mkdir -p ~/.ssh && \
        umask 0077 && \
        echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa && \
        git config --global url."git@bitbucket.org:".insteadOf https://bitbucket.org/ && \
        ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts

RUN \
        go get && \
        CGO_ENABLED=0 go build -o app

# final stage of multi-stage: will appropriately *NOT* include SSH keys
FROM scratch

COPY --from=build \
        /etc/ssl /etc/ssl

COPY --from=build \
    /bld/app /app/myapp

CMD ["/app/myapp"]