How to view full dependency tree for nested Go dependencies

Question

I'm trying to debug the following build error in our CI where "A depends on B which can't build because it depends on C." I'm building my data service which doesn't directly depend on kafkaAvailMonitor.go which makes this error hard to trace. In other words:

data (what I'm building) depends on (?) which depends on kafkaAvailMonitor.go

It may seem trivial to fix for a developer they just do "go get whatever" but I can't do that as part of the release process - I have to find the person that added the dependency and ask them to fix it.

I'm aware that there are tools to visualize the dependency tree and other more sophisticated build systems, but this seems like a pretty basic issue: is there any way I can view the full dependency tree to see what's causing the build issue?

go build -a -v

../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package 
  "github.com/Shopify/sarama/tz/breaker" in any of:
  /usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker

Answer 1

go mod graph and https://github.com/vc60er/deptree can only handle go mod, but NOT package.

I find the tool depth can handle package dependency tree in command line.

And it supports -explain to show the package import chain, like go mod why.

https://github.com/KyleBanks/depth depth is tool to retrieve and visualize Go source code dependency trees.

If you want show the dependencies in dot graph, try godepgraph. It runs faster but you need dot installed.

https://github.com/kisielk/godepgraph godepgraph is a program for generating a dependency graph of Go packages.

Answer 2

can try this https://github.com/vc60er/deptree

 redis git:(master) go mod graph | deptree -d 3
package: github.com/go-redis/redis/v9
dependence tree:

┌── github.com/cespare/xxhash/v2@v2.1.2
├── github.com/dgryski/go-rendezvous@v0.0.0-20200823014737-9f7001d12a5f
├── github.com/fsnotify/fsnotify@v1.4.9
│    └── golang.org/x/sys@v0.0.0-20191005200804-aed5e4c7ecf9
├── github.com/nxadm/tail@v1.4.8
│    ├── github.com/fsnotify/fsnotify@v1.4.9
│    │    └── golang.org/x/sys@v0.0.0-20191005200804-aed5e4c7ecf9
│    └── gopkg.in/tomb.v1@v1.0.0-20141024135613-dd632973f1e7
├── github.com/onsi/ginkgo@v1.16.5
│    ├── github.com/go-task/slim-sprig@v0.0.0-20210107165309-348f09dbbbc0
│    │    ├── github.com/davecgh/go-spew@v1.1.1
│    │    └── github.com/stretchr/testify@v1.5.1
│    │         └── ...

Answer 3

The above answer still doesn't show me a dependency tree so I've taken the time to write a Python script to do what I need - hopefully that helps other people.

The issue with the above solution (the others proposed like go list) is that it only tells me the top level. They don't "traverse the tree." This is the output I get - which doesn't help any more than what go build gives me.

.../npd/auth/
   .../mon/mlog
   .../auth/service

This is what I'm trying to get - I know that auth is broken (top) and that breaker is broken (bottom) from go build but I have no idea what's in between - my script below gives me this output.

.../npd/auth/
    .../npd/auth/service                
        .../npd/auth/resource
            .../npd/auth/storage
               .../npd/middleware
                  .../npd/metrics/persist
                    .../npd/kafka
                        .../vendor-library/src/github.com/Shopify/sarama
                            .../vendor-library/src/github.com/Shopify/sarama/vz/breaker

My Python script:

import subprocess
import os

folder_locations=['.../go/src','.../vendor-library/src']

def getImports(_cwd):
    #When the commands were combined they overflowed the bugger and I couldn't find a workaround
    cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
    cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]

    process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)

    out1, err = process.communicate()

    process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)

    out2, err = process.communicate()
    out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')

    return str(out1).split('\\n'),out2clean.split('\\n')

def getFullPath(rel_path):
    for i in folder_locations:
        if os.path.exists(i+'/'+rel_path):
            return i+'/'+rel_path
    return None

def getNextImports(start,depth):

    depth=depth+1
    indent = '\t'*(depth+1)

    for i,val in enumerate(start.keys()):

        if depth==1:
            print (val)

        out1,out2=getImports(val)

        noDeps=True
        for j in out2[i].split(' '):
            noDeps=False
    
            _cwd2=getFullPath(j)
            new_tree = {_cwd2:[]}
            not_exists = (not _cwd2 in alltmp)
    
            if not_exists:
                print(indent+_cwd2)
                start[val].append(new_tree)
                getNextImports(new_tree,depth)
                alltmp.append(_cwd2)

        if noDeps:
            print(indent+'No deps')

_cwd = '/Users/.../npd/auth'

alltmp=[]
start_root={_cwd:[]}
getNextImports(start_root,0)

Answer 4

When using modules you may be able to get what you need from go mod graph.

usage: go mod graph

Graph prints the module requirement graph (with replacements applied)
in text form. Each line in the output has two space-separated fields: a module
and one of its requirements. Each module is identified as a string of the form
path@version, except for the main module, which has no @version suffix.

I.e., for the original question, run go mod graph | grep github.com/Shopify/sarama then look more closely at each entry on the left-hand side.

Answer 5

I just want to mention here that go mod why can also help. Anyway you cannot get and display the whole tree. But you can trace back one single branch of a child dependency until its parent root.

Example:

$ go mod why github.com/childdep
# github.com/childdep
github.com/arepo.git/service
github.com/arepo.git/service.test
github.com/anotherrepo.git/mocks
github.com/childdep

That means, you have imported 'childdep' finally in 'anotherrepo.git/mocks'.

Answer 6

if the following isn't a stack trace what is it?

It is the list of path where Go is looking for your missing package.

I have no idea who is importing kafkaAvailMonitor.go

It is not "imported", just part of your sources and compiled.
Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.

Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.

go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run

You can then script go list in order to list all dependencies.
See this bash script for instance, by Noel Cower (nilium)

#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports).  You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.

: "${TESTIMPORTS:=1}"

lsdep_impl__ () {
    local txtestimps='{{range $v := .TestImports}}{{print . "\n"}}{{end}}'
    local txdeps='{{range $v := .Deps}}{{print . "\n"}}{{end}}'

    {
        go list -f "${txtestimps}${txdeps}" "$@"
        if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
        then
            go list -f "${txtestimps}" "$@" |
            sort | uniq |
            comm -23 - <(go list std | sort) |
                TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
        fi
    } |
    sort | uniq |
    comm -23 - <(go list std | sort)
}
export -f lsdep_impl__

lsdep_impl__ "$@"