CODEWITHSUNDEEP
gogo-modulesgo-build
Feeble Wildcat
Feeble Wildcat

Reputation: 111

Pre-compiling Golang project dependencies to cache

In short, my current use-case involves dynamically creating a Golang plugin inside a Docker container. The compilation involves some new input from the user (which is why it is not compiled beforehand), but the dependencies are static, and won't change.

Currently, the full compilation is done from scratch inside the Docker container (though go mod download is used to reduce the time by a bit). I noticed that the go build command ends up compiling a lot of the dependencies, which adds a non-trivial amount of time for the plugin compilation, which affects the usability of my application.

Is there a Go supported method or command to read through the go.mod file and populate the GOCACHE directory? With such a command, I would run it in my Dockerfile itself, causing the Docker image to contain the cache with all the compiled build dependencies.

What I've tried:

  1. go mod download: This only downloads the dependencies; it does not compile them.
  2. I do have this working with a temporary workaround: I created a barebones main.go that imports all the dependencies, and run go build within my Dockerfile to populate the cache. As mentioned, this does solve my problem, but it feels like a bit of a hack. Additionally, if the dependencies change in the future, it requires someone to change this as well, which isn't ideal.
  3. A lot of the answers I saw online for this involve CI/CD. With CI/CD, the container just has a partition mounted to the host, which contains a cache that is persisted after runs. This does not solve my immediate problem, which is for building the container itself.

Upvotes: 10

Views: 1468

Answers (2)

CodePerson
CodePerson

Reputation: 1

I do have this working with a temporary workaround: I created a barebones main.go that imports all the dependencies, and run go build within my Dockerfile to populate the cache. As mentioned, this does solve my problem, but it feels like a bit of a hack.

^^ I think your "temporary workaround" is the simplest way to cache compiled 3rd party + transitive dependencies. Below is my recommended implementation:

Steps:

  1. Create a dependencies.go file, preferably at the root directory, like mine below, which includes packages from your go.mod file:

dependencies.go

// depedencies.go

package main

import (
    _ "runtime/pprof"
    _ "testing"
    _ "github.com/Knetic/govaluate"
    _ "github.com/alecthomas/chroma/v2"
    _ "github.com/cloudflare/cfssl/helpers"
    _ "github.com/go-faker/faker/v4"
    _ "github.com/go-resty/resty/v2"
    _ "github.com/go-rod/rod"
    _ "github.com/google/certificate-transparency-go"
    _ "github.com/hashicorp/go-version"
    _ "github.com/jaytaylor/html2text"
    _ "github.com/kataras/jwt"
    _ "github.com/kljensen/snowball"
    _ "github.com/kljensen/snowball/english"
    _ "github.com/schollz/progressbar/v3"
    _ "github.com/shirou/gopsutil/v3/mem"
    _ "github.com/spf13/cobra"
    _ "github.com/spf13/viper"
    _ "github.com/tklauser/go-sysconf"
    _ "github.com/yuin/goldmark/text"
    _ "github.com/zcalusic/sysinfo"
    _ "golang.org/x/crypto/openpgp/packet"
    _ "golang.org/x/crypto/scrypt"
    _ "golang.org/x/net/context"
    _ "golang.org/x/text/cases"
    _ "golang.org/x/text/encoding/korean"
    _ "golang.org/x/text/encoding/traditionalchinese"
    _ "golang.org/x/text/language"
    _ "golang.org/x/time/rate"
    _ "gopkg.in/yaml.v2"
)

func main() {
    
}
  1. Make a Dockerfile like below, in the root directory.

Dockerfile

FROM --platform=linux/amd64 golang:1.22.1-bullseye as golang-image-builder
WORKDIR /src

COPY go.mod .
COPY go.sum .
RUN go mod download

COPY dependencies.go .
RUN go build all
RUN rm dependencies.go

COPY config ./config
COPY cmd ./cmd
COPY internal ./internal

ENV VERSION="1.2.3"
RUN go build -v -o executable-name -ldflags="-s -w -X main.version=$VERSION" ./cmd/executable-name

ENTRYPOINT ["./executable-name"]
  1. Pay attention to the output of RUN go build -v -o executable-name -ldflags="-s -w -X main.version=$VERSION" ./cmd/executable-name, which will likely tell of you names of extra packages you should add to your dependencies.go file

Upvotes: 0

Paramtamtаm
Paramtamtаm

Reputation: 340

Since the issues (1, 2) in the golang repository are still open, all that we can do is "hacking", I think. So, we can do something like that for the dependencies caching and pre-compilation as a separate docker layer:

FROM golang:1.19-buster as builder

COPY ./go.* /src/

WORKDIR /src

# burn the modules cache
RUN set -x \
    # cache go dependencies
    && go mod download \
    # pre-compile common dependencies
    && mkdir /tmp/gobin \
    && for p in $(go list -m -f '{{if and (not .Indirect) (not .Main)}}{{.Path}}/...@{{.Version}}{{end}}' all); do \
      GOBIN=/tmp/gobin go install $p; \
    done \
    && rm -r /tmp/gobin

COPY . /src

RUN go build ...

Without this trick, the building (docker buildx build --platform linux/amd64,linux/arm64 ...) takes about 9 minutes, with it ~6 minutes (profit 30%). But the pre-compilation step becomes longer by ~40%.

Upvotes: 3

Related Questions