Golang build all downloaded packages

Question

I'm trying to build a docker image for my go application and I'd like to build all of the package dependencies separately from compiling the source code of my application.

Here's an example that might be helpful:

FROM ubuntu:focal

# ... a bunch of run commands that install go and other binaries I need

ENV GOPATH=/root/go
WORKDIR gadic

COPY go.mod .
RUN go mod download
RUN go build /root/go/pkg/... # this fails!!

COPY src src
RUN go build path/to/main.go # I'd like to speed this step up by pre-compiling the packages main.go depends on

When I try to build this dockerfile I get the following output:

 > [ 8/19] RUN go build /root/go/pkg/...:
#12 0.955 pattern /root/go/pkg/...: directory prefix /root/go/pkg outside available modules

Is there a way to build the packages/modules that my application depends on prior to building the main.go of my application?

Answer 1

What you really need is a build cache, see this.

There are two folders needed to be mounted into build container:

• GOPATH

  Here we set as /root/go, used to store downloaded dependency

• GOCACHE

  Default is /root/.cache/go-build, used for build cache

A minimal example as next.

Dockerfile:

# syntax = docker/dockerfile:1.2
FROM golang
ENV GOPATH=/root/go
WORKDIR /root
COPY main.go .
COPY go.mod .
COPY go.sum .

RUN --mount=type=cache,mode=0755,target=/root/.cache/go-build --mount=type=cache,mode=0755,target=/root/go go build -v main.go

main.go:

package main

import (
     _ "github.com/phachon/go-logger"
)

func main() {
}

go.mod:

module trial

go 1.14

require github.com/phachon/go-logger v0.0.0-20191215032019-86e4227f71ea

go.sum:

github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/mailru/easyjson v0.7.0 h1:aizVhC/NAAcKWb+5QsU1iNOZb4Yws5UO2I+aIprQITM=
github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.11 h1:FxPOTFNqGkuDUGi3H/qkUbQO4ZiBa2brKq5r0l8TGeM=
github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE=
github.com/phachon/go-logger v0.0.0-20191215032019-86e4227f71ea h1:IkOONr/u7Wy+j2R4r1eMV8PEuN4kmOhZZNaYxDOF+KQ=
github.com/phachon/go-logger v0.0.0-20191215032019-86e4227f71ea/go.mod h1:WBIWFH/iYYvuApCvPU+/R6hfX6v0Ogu4apwf0UgzVF0=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193GlqGZbnPFnPV/5Rsb4=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=

Execution:

• 1st time, you could see it downloads the dependency package, and then build different packages.

$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
...
#14 [stage-0 6/6] RUN --mount=type=cache,mode=0755,target=/root/.cache/go-bu...
#14 3.100 go: downloading github.com/phachon/go-logger v0.0.0-20191215032019-86e4227f71ea
#14 3.330 go: downloading github.com/fatih/color v1.7.0
#14 3.331 go: downloading github.com/mailru/easyjson v0.7.0
#14 3.523 go: downloading github.com/mattn/go-colorable v0.1.4
#14 3.561 go: downloading github.com/mattn/go-isatty v0.0.11
#14 3.731 go: downloading golang.org/x/sys v0.0.0-20191026070338-33540a1f6037
#14 5.058 github.com/mailru/easyjson/buffer
#14 5.063 golang.org/x/sys/unix
#14 5.070 github.com/mailru/easyjson/jlexer
#14 5.077 github.com/mailru/easyjson/jwriter
#14 5.087 github.com/phachon/go-logger/utils
#14 5.174 github.com/mailru/easyjson
#14 5.346 github.com/mattn/go-isatty
#14 5.355 github.com/mattn/go-colorable
#14 5.368 github.com/fatih/color
#14 5.396 github.com/phachon/go-logger
#14 5.451 command-line-arguments
#14 DONE 7.4s
...

• 2nd time, I changed the main.go as next:

main.go:

package main

import (
     _ "github.com/phachon/go-logger"
     "fmt"
)

func main() {
    fmt.Println("helloworld")
}

Then, execute build again:

$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
...
#14 [stage-0 6/6] RUN --mount=type=cache,mode=0755,target=/root/.cache/go-bu...
#14 1.469 command-line-arguments
#14 DONE 3.2s
...

You could see from above, it no longer downloads package (because /root/go was reused during different times build). And, it just build command-line-arguments, corresponding to the change of main.go, no packages related to github.com/phachon/go-logger been rebuilt (because /root/.cache/go-build as build cache be reused).

Answer 2

I think you should take the advantage of the multi-stage builds. In the first stage, you use the docker image with Go installed. You can use one of those available here https://hub.docker.com/_/golang. At this point, you have to copy your source code into the container and run go build there.

In the next step, you (for example) use an alpine image and copy the executable (and other required fields) to the target container.

This is an example how it can be done.

FROM golang:alpine AS build-env
RUN apk --no-cache add build-base git bzr mercurial gcc
ADD . /src
RUN cd /src && go build -o goapp

# final stage
FROM alpine
WORKDIR /app
COPY --from=build-env /src/goapp /app/
ENTRYPOINT ./goapp

This has several benefits and the size of the final image is one of them.