Getting all dependencies of a Go project

Question

For my project I am trying to get all the dependencies and sub dependencies of my project. I need to specific version of each of these dependencies. Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.

For my project, go list -m all works for everything except indirect dependencies that have not opted into using go.mod files. Right now my workflow is taking an initial batch of repositories, downloading them from git then using “GO111MODULE=on go build ./…”. and “GO111MODULE=on go list -m -json all” to get the list of dependencies. I do not check for go.mod as all of the repositories I am scanning are using go.mod files.

For the list of dependencies that come out of this initial list I have some questions, for files without go.mod files, I used this as a reference: “https://blog.golang.org/using-go-modules”

-Path = Received from go list -m all, it can be GitHub, gopkg, or whatever is used to dl the go package.

Without go.mod

-“GO111MODULE=on go mod init <PATH from parent go.mod>”

-“GO111MODULE=on go build ./…”

-“GO111MODULE=on go mod tidy”

-“GO111MODULE=on go list -m -json all”

-From there I get a list of the dependencies of this module. 

With go.mod

-“GO111MODULE=on go build ./…”

-“GO111MODULE=on go mod tidy”

-“GO111MODULE=on go list -m -json all”

Should I be running go build on each dependency that has a go.mod file? For ones without a go.mod file, I understand this should be done, as how else will we populate the go.mod file with the dependencies. But for files with a go.mod file, will I be pulling extra stuff that is not necessarily being used by my project with go build, like test files and other files that might not be used when I am simply importing that project? I understand that its better to get more unused dependencies rather than missing some, but it is getting a bit overwhelming with how massive the amount of dependencies is.

Answer 1

Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.

For compiling or running your project?
Because you also have tools, which don't directly build or run your project, but assist your source generation or static analysis (linter).

For that, you have a proposal for "Adding tool dependencies to go.mod".
It is accepted and implemented in golang/go issue 48429.
You will have go get -tool example.com/m1 to add a tool line to your mod file and add any missing dependencies.

And when go tool is run in module mode with an argument that does not match a Go built-in tool, it will search the current for a tool directive that matches the last path segment. It will then compile and run that tool similarly to go run.

---

As mentioned in the FAQ:

in Go 1.24 and above, you can add a tool directive to your go.mod:

go 1.24

...

tool golang.org/x/tools/cmd/stringer

Before Go 1.24 the recommended approach was to add a tools.go file to your module that includes import statements for the tools of interest (such as import _ "golang.org/x/tools/cmd/stringer"), along with a //go:build tools build constraint.
The import statements allow the go command to precisely record the version information for your tools in your module's go.mod, while the //go:build tools build constraint prevents your normal builds from actually importing your tools.

Same in Managing Dependencies / Tool dependencies or in Modules:

The tool meta-pattern provides a way to perform operations on all tools simultaneously.
For example, you can upgrade all tools with go get -u tool, or install them all to $GOBIN with go install tool.

In Go versions before 1.24, you can achieve something similar to a tool directive by adding a blank import to a go file within the module that is excluded from the build using build constraints.
If you do this, you can then use go run with the full package path to run the tool.

The tool caching is nice:

Executables created by go run and the new behavior for go tool are now stored in the go cache.
This makes repeated executions faster at the expense of making the Go cache larger.
These executables will be cleaned up automatically, but if you need to reclaim space, you can run go cache -clean manually. See #69290.

Answer 2

The phrase “all the dependencies” is unfortunately ambiguous.

go list -m lists all modules whose selected versions are determined by your go.mod file. That is one possible definition of "all the dependencies", but it is a much broader set of modules than I think most people intend when they talk about the “dependencies” of a module.

In practice, go list -test all (without the -m flag) is the broadest set of dependencies I would generally care about: it includes all of the packages listed in import statements within your module (i.e. everything you need in order to run go test ./... within your module), plus all of the packages transitively needed to run go test on those packages.

(In particular, go list -test all is also the set of packages that will be resolved when you run go mod tidy.)

Answer 3

I can try to analyze go.sum file (when you execute go list -u, go.sum was created)

The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control. (Using Go Modules - Adding a dependency)

go.sum file lists down the checksum (and version tag) of direct and indirect dependency required by the module.

% cat go.sum
...
github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q=
github.com/c2h5oh/datasize v0.0.0-20171227191756-4eba002a5eae/go.mod h1:S/7n9copUssQ56c7aAgHqftWO4LTf4xY6CGWt8Bc+3M=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
...