Git returns 'failed to run repack' and 'inflate returned' errors

Question

I am facing a problem with a Git repository stored on GitLab. It seems to be a repository issue only affecting this specific repository, as all other projects hosted on GitLab are working fine.

It seems I can personally push, pull, and checkout branches using GitKraken, but when I try to pull from Git Bash I get the following:

$ git pull
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: Could not read bb5a805503a3da247038200df7002f452a8781e9
fatal: bad tree object bb5a805503a3da247038200df7002f452a8781e9
error: failed to run repack

All people cooperating with me on this same project have similar issues when attempting to pull:

$ git pull
remote: Enumerating objects: 112, done.
remote: Counting objects: 100% (112/112), done.
remote: Compressing objects: 100% (102/102), done.
fatal: pack has bad object at offset 8105548: inflate returned -5
fatal: index-pack failed

And this is what we all get when trying to clone the repository on a new location:

remote: Enumerating objects: 4364, done.
remote: Counting objects: 100% (4364/4364), done.
remote: Compressing objects: 100% (1622/1622), done.
fatal: pack has bad object at offset 56589977: inflate returned -5
fatal: index-pack failed

These are the things we tried:

• Tested multiple computers
• Tested multiple users

I suspect, reading similar questions such as this one, that our repository is broken. However, I don't understand why I seem to be able to work with it via GitKraken. With its GUI, I successfully merged two branches and pushed the latest commits to the server.

Does anybody have an explanation of what could be the problem?

Edit: attempt of fixing the repository

Following these instructions contained in the link I posted, and also suggested by an answer below, I ran the command git fsck --full to examine the state of the repository links. What I found is not reassuring, as it seems many links are broken.

Checking object directories: 100% (256/256), done.
Checking objects: 100% (3769/3769), done.
broken link from  commit f42ccacb8101ef49493aca18089378697490bb66
              to    tree e461e3cbe3221cd5ba7035222aa716dcabb63713
broken link from  commit 2fe8ac2b06d8e8f37b354c395f60a77f0ab1f9a9
              to    tree 93b9618cc159c1b18aba319e8f7e3e5e8f7b57df
broken link from  commit 16d23305969b3a40316618b952b2e5ff1ffedbf6
              to    tree 80c4012d9f3b3f51f17932dec80e740bc4e5a1d6
broken link from    tree 867941d734b41a5ce800dff6ea7dbfca30787e15
              to    tree bb5a805503a3da247038200df7002f452a8781e9
broken link from    tree e16211709ea4ce02a89bbe87d30a410dac65e372
              to    blob b6eb83a9e4f16fe49a0eb9bfea0bf6dfce9adcbc
broken link from    tree e16211709ea4ce02a89bbe87d30a410dac65e372
              to    blob a593c8f43faacf41bc93c98dbb347e673cd47f3f
broken link from    tree e16211709ea4ce02a89bbe87d30a410dac65e372
              to    blob 652245900beb49246e58f5c216dbcf161f727e2d
broken link from    tree e16211709ea4ce02a89bbe87d30a410dac65e372
              to    blob a7998441f7435126feb6b35e9b4b575bd193d6d2

followed by a long list of dangling commit and dangling blob lines with 8 instances of missing ones:

[...]
missing tree 80c4012d9f3b3f51f17932dec80e740bc4e5a1d6
[...]
missing blob a593c8f43faacf41bc93c98dbb347e673cd47f3f
[...]
[6 more]

It seems like a manual recovery of the broken files would take quite some time.

Edit #2: fixed repository locally

I installed and ran git-repair in my local copy of the repository, and actually got it fixed. If I now run git fsck --full I only see "healthy" messages in response. No more broken links.

However, no matter if I git push --force to origin, it seems origin remains broken. One weird update is that now I can use git clone successfully, while all my co-workers still cannot. How can it be?

And most importantly, is there a way I can actually run git-repair on origin?

Edit #3: Made new origin

After fixing my repository locally (and checked that git fsck gives no missing links) I pushed all relevant branches to a fresh origin on GitLab. I thought that was going to be it, but unfortunately the problem persists.

A pattern I start to notice is that we seem to all be able to clone from Ubuntu (either using Git Bash or GitKraken) but not on Windows 10 (neither using Git Bash nor GitKraken).

Reading around the website I found a question about how could it be possible that Git worked on Linux but not on Windows. There they explained it was a Git-related issue (but it was more than 1 year ago). Would it make sense something similar has happened? I have to say I'm skeptical about it, because other repositories we tested with work fine on Windows.

Edit #4: tested with older versions of Git for Windows

Current version is 2.19. I tried it on 2.18 and 2.9 (the latter dating 2016) but I get the same error.

Edit #5: tried cloning locally successfully

After a suggestion on a GitHub issue I wrote on git-for-windows, I tried to use git clone from a copy of the repository on a USB stick. It worked. The problem seems to be confined to either Git+Windows+GitLab or Git+Windows+SSH.

Answer 1

A pattern I start to notice is that we seem to all be able to clone from Ubuntu (either using Git Bash or GitKraken) but not on Windows 10 (neither using Git Bash nor GitKraken).

That could be explained by the Git version (sometimes 1.18 on old Ubuntu, versus 2.x on Windows)

Git v2.3.5 (Q1 2015) introduced a "ref paranoia" to prevent that kind of clone (of a broken remote repository)

refs: introduce a "ref paranoia" flag

Most operations that iterate over refs are happy to ignore broken cruft.
However, some operations should be performed with knowledge of these broken refs, because it is better for the operation to choke on a missing object than it is to silently pretend that the ref did not exist (e.g., if we are computing the set of reachable tips in order to prune objects).

These processes could just call for_each_rawref, except that ref iteration is often hidden behind other interfaces.
For instance, for a destructive "repack -ad", we would have to inform "pack-objects" that we are destructive, and then it would in turn have to tell the revision code that our "--all" should include broken refs.

It's much simpler to just set a global for "dangerous" operations that includes broken refs in all iterations.

You had in Documentation/git.txt

GIT_REF_PARANOIA::

If set to 1, include broken or badly named refs when iterating over lists of refs.

In a normal, non-corrupted repository, this does nothing.

However, enabling it may help git to detect and abort some operations in the presence of broken refs.

Git sets this variable automatically when performing destructive operations like git prune.
You should not need to set it yourself unless you want to be paranoid about making sure an operation has touched every ref (e.g., because you are cloning a repository to make a backup).

---

With Git 2.34 (Q4 2021), the ref iteration code used to optionally allow dangling refs to be shown, and it has been tightened up.

The clone on Ubuntu with the latest Git won't succeed!

See commit 67985e4, commit 2d653c5, commit 1763334, commit 5d1f5b8, commit 968f12f, commit 6d751be, commit 8dccb22, commit 9aab952, commit bf708ad, commit 5b062e1, commit 078eecb, commit f805844, commit 2ac0cbc, commit da5e0c6, commit e9de7a5, commit b472424 (24 Sep 2021) by Jeff King (peff).
^{(Merged by Junio C Hamano -- gitster -- in commit f6c075a, 11 Oct 2021)}

refs: turn on GIT_REF_PARANOIA by default

^{Signed-off-by: Jeff King
Reviewed-by: Jonathan Tan}

The original point of the GIT_REF_PARANOIA flag was to include broken refs in iterations, so that possibly-destructive operations would not silently ignore them (and would generally instead try to operate on the oids and fail when the objects could not be accessed).

We already turned this on by default for some dangerous operations, like "repack -ad" (where missing a reachability tip would mean dropping the associated history).
But it was not on for general use, even though it could easily result in the spreading of corruption (e.g., imagine cloning a repository which simply omits some of its refs because their objects are missing; the result quietly succeeds even though you did not clone everything!).

This patch turns on GIT_REF_PARANOIA by default.

So a clone as mentioned above would actually fail (upload-pack tells us about the broken ref, and when we ask for the objects, pack-objects fails to deliver them).

This may be inconvenient when working with a corrupted repository, but:

• we are better off to err on the side of complaining about corruption, and then provide mechanisms for explicitly loosening safety.
• this is only one type of corruption anyway.

If we are missing any other objects in the history that aren't ref tips, then we'd behave similarly (happily show the ref, but then barf when we started traversing).

We retain the GIT_REF_PARANOIA variable, but simply default it to "1" instead of "0".
That gives the user an escape hatch for loosening this when working with a corrupt repository.
It won't work across a remote connection to upload-pack (because we can't necessarily set environment variables on the remote), but there the client has other options (e.g., choosing which refs to fetch).

As a bonus, this also makes ref iteration faster in general (because we don't have to call has_object_file() for each ref), though probably not noticeably so in the general case.
In a repo with a million refs, it shaved a few hundred milliseconds off of upload-pack's advertisement; that's noticeable, but most repos are not nearly that large.

The possible downside here is that any operation which iterates refs but doesn't ever open their objects may now quietly claim to have X when the object is corrupted (e.g., "git rev-list new-branch --not --all"^(man)will treat a broken ref as uninteresting).
But again, that's not really any different than corruption below the ref level.
We might have refs/heads/old-branch as non-corrupt, but we are not actively checking that we have the entire reachable history.
Or the pointed-to object could even be corrupted on-disk (but our "do we have it" check would still succeed).
In that sense, this is merely bringing ref-corruption in line with general object corruption.

One alternative implementation would be to actually check for broken refs, and then immediately die if we see any.
That would cause the "rev-list --not --all^(man) case above to abort immediately.
But in many ways that's the worst of all worlds:

• it still spends time looking up the objects an extra time
• it still doesn't catch corruption below the ref level
• it's even more inconvenient; with the current implementation of GIT_REF_PARANOIA for something like upload-pack, we can make the advertisement and let the client choose a non-broken piece of history.
  If we bail as soon as we see a broken ref, they cannot even see the advertisement.

git now includes in its man page:

If set to 0, ignore broken or badly named refs when iterating over lists of refs.
Normally Git will try to include any such refs, which may cause some operations to fail.

This is usually preferable, as potentially destructive operations (e.g., git prune) are better off aborting rather than ignoring broken refs (and thus considering the history they point to as not worth saving).

The default value is 1 (i.e., be paranoid about detecting and aborting all operations). You should not normally need to set this to 0, but it may be useful when trying to salvage data from a corrupted repository.

Answer 2

As the linked answer points out, most of the objects in the tree are diffed and compressed to save bandwidth and disk, creating the mentioned .pack files.

During the pull process, the git client exchanges with the remote a list of refs and heads, so the remote can walk them and compute what objects need to be sent based on what the client already has. It then creates the packs And sends them to the client. This means that a pack sent to a client could be different from another for the same object.

As for why, I couldn't tell, but it's plausible that your repo is skipping the conflicting object, although since different clients throw this error in the same working copy this seems unlikely and looks like it might be just a software issue, maybe differences in the zlib used by each client.

You could try to force-push your repo (not recommended) or pushing to other remote as a last resort.

Also, from the answer you link, there's a procedure to try and fix broken repositories.

UPDATE: As a followup, -5 is Z_BUF_ERROR on zlib and according to the documentation:

inflate() returns Z_OK if some progress has been made [...] Z_BUF_ERROR if no progress was possible or if there was not enough room in the output buffer when Z_FINISH is used. Note that Z_BUF_ERROR is not fatal, and inflate() can be called again with more input and more output space to continue decompressing. If Z_DATA_ERROR is returned[...]

I personally think the bug is in the software, rather than in GitLab, but that's speculation on my part.

And a suggestion: If it's not possible to move all devs to Linux... What about the Linux subsystem for W10 ?