How do I debug `cargo build` hanging at "Updating crates.io index"?

Question

My Rust project no longer builds after some conflicts between my RLS plugin and terminal build. I searched around the web and found suggestion of removing my ~/.cargo/registry/index/*, but after that I can't even build any project.

Now the build always stops at

Updating crates.io index

Passing in the --verbose option doesn't help so I don't even know whether it's dying. What should I do next? How to debug this issue?

Answer 1

I would recommend not wiping any folders, as the active data is a git repository with some cryptographic registry data, and the common cause is network issues communicating with GitHub. Whether the problem is simply a slow connection or worsely compromised infrastructure, wiping the folder could remove correct data you have already downloaded that would then be redownloaded through the nonresponsive link. If the problem is local data corruption, then wiping the folder would make sense.

It sounds like cargo is missing an important logging facility for git network activity, unless somebody more familiar with it knows further.

Here is how I diagnosed this without knowledge of cargo. It seems it would be faster if the user knew about the cargo registry.

# ran failing process in background
$ cargo update &
     Updating crates.io index
[1] 8129

# store PID of background process in variable
$ CARGO_PID=$1

# listed files accessed by process
# the one of note is the git pack file
$ ls -l /proc/$CARGO_PID/fd
total 0
lrwx------. 1 user user 64 Nov  7 06:19 0 -> /dev/pts/377
lrwx------. 1 user user 64 Nov  7 06:19 1 -> /dev/pts/377
lrwx------. 1 user user 64 Nov  7 06:19 10 -> 'socket:[45370067]'
lrwx------. 1 user user 64 Nov  7 06:19 11 -> 'socket:[45383920]'
lrwx------. 1 user user 64 Nov  7 06:19 2 -> /dev/pts/377
lrwx------. 1 user user 64 Nov  7 06:19 3 -> /home/user/.cargo/.package-cache
lr-x------. 1 user user 64 Nov  7 06:19 4 -> /home/user/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack-4bac04bfb4c5de794c1dd0a40d86cfc18f9eb5ae.pack
lrwx------. 1 user user 64 Nov  7 06:19 5 -> 'socket:[45370061]'
lr-x------. 1 user user 64 Nov  7 06:19 6 -> /home/user/.pki/nssdb/cert9.db
lr-x------. 1 user user 64 Nov  7 06:19 7 -> /home/user/.pki/nssdb/key4.db
lr-x------. 1 user user 64 Nov  7 06:19 8 -> /etc/pki/nssdb/cert9.db
lr-x------. 1 user user 64 Nov  7 06:19 9 -> /etc/pki/nssdb/key4.db

# it's working with a git repository, visited git repository
$ cd /home/user/.cargo/registry/index/github.com-1ecc6299db9ec823/.git

# i then looked various common places for the remote it was fetching from.
# FETCH_HEAD showed this in one run, but in another FETCH_HEAD was empty
# I then found it in the logs folder.
# you could also use a network utility, strace, or debug or patch cargo
$ cat config
[core]
    bare = false
    repositoryformatversion = 0
    filemode = true
    logallrefupdates = true
$ ls
config  description  FETCH_HEAD  HEAD  hooks  info  logs  objects  refs
$ cat HEAD
ref: refs/heads/main
$ cat FETCH_HEAD
$ ls logs
refs
$ ls logs/refs/
remotes
$ ls logs/refs/remotes/
origin
$ ls logs/refs/remotes/origin/
HEAD
$ cat logs/refs/remotes/origin/HEAD
0000000000000000000000000000000000000000 cd2e0d3249278088bdb2c8ab9227c1e745977d2a John Doe <johndoe@example.com> 1671547379 -0500   fetch https://github.com/rust-lang/crates.io-index
cd2e0d3249278088bdb2c8ab9227c1e745977d2a 995b889318272a6dbc9591a688fc6d0053fbdacf John Doe <johndoe@example.com> 1671550295 -0500   fetch https://github.com/rust-lang/crates.io-index
995b889318272a6dbc9591a688fc6d0053fbdacf e6ca1cbd89a020cfccfb741738b47d905ec51a38 John Doe <johndoe@example.com> 1671965624 -0500   fetch https://github.com/rust-lang/crates.io-index
e6ca1cbd89a020cfccfb741738b47d905ec51a38 9dfcadd951ddd8b5b311b0653bf09069b62594a3 John Doe <johndoe@example.com> 1672045603 -0500   fetch https://github.com/rust-lang/crates.io-index
9dfcadd951ddd8b5b311b0653bf09069b62594a3 2df4156cfe1fba11c11f8b1956ac8c3012da94da John Doe <johndoe@example.com> 1672051948 -0500   fetch https://github.com/rust-lang/crates.io-index
2df4156cfe1fba11c11f8b1956ac8c3012da94da 53c0024ba8565070cc0644cc4417d81e2d588aac John Doe <johndoe@example.com> 1672324569 -0500   fetch https://github.com/rust-lang/crates.io-index
53c0024ba8565070cc0644cc4417d81e2d588aac 2bdf23bbb3e8e1752ab88485168330751d8d9e49 John Doe <johndoe@example.com> 1683666817 -0400   fetch https://github.com/rust-lang/crates.io-index

Now (if you scroll way to the bottom right above) one can see that it is likely hanging performing a git fetch from https://github.com/rust-lang/crates.io-index.

# terminate background process
$ fg
cargo update    (wd: ~/src/project)
^C

# try fetching from its url
$ git fetch --progress --verbose https://github.com/rust-lang/crates.io-index
POST git-upload-pack (117 bytes)
POST git-upload-pack (958 bytes)
POST git-upload-pack (967 bytes)
remote: Enumerating objects: 626476, done.
remote: Counting objects: 100% (35613/35613), done.
remote: Compressing objects: 100% (1310/1310), done.
Receiving objects:  98% (615100/626476), 399.19 MiB | 1.36 MiB/s
# hangs at 98%, no more output

The troubleshooting has simplified to diagnosing a git fetch from this repository.

Personally, my dmesg then started spewing errors regarding my wifi driver, perhaps an unfortunate coincidence or alternatively an indication of malicious activity. More commonly, my available disk space was getting very low, and I was having network issues communicating with github.com .

error: RPC failed; curl 56 I/O operation timed out
error: 4611 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Answer 2

Delete the ~/.cargo/.package-cache file.

Answer 3

Change cargo source in ~/.cargo/config. This works for me.

For example:

[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = 'ustc'

[source.ustc]
registry = "git://mirrors.ustc.edu.cn/crates.io-index"