How to fix a git repository with lots of dangling objects, that fails git gc --prune=now with fatal: bad object:

Question

I have a repository whose remote is on bitbucket. At certain point I pushed a branch to server, and by mistake asked to push all local tags.

The action failed after some time with an error (cannot recall exactly - but I think I was not allowed to change tags on the server).

However - since then I have these problems.

1. lots (~130) of dangling blobs, commits and tags

2. Any attempt to rid of these "dangling" objects - fails with lots of errors, all pointing to one specific "bad object"

    mini-mac7:myRepo $ git fsck
    Checking object directories: 100% (256/256), done.
    Checking objects: 100% (16253/16253), done.
    broken link from  commit c521a9214898aac031831d97e6a8329342b788c4
                  to  commit a2df338e00be26da2f0b84543668ef29b52df5c3
    broken link from  commit 19f55b6374e7adcc42046821c1659bb47df933a8
                  to  commit 0d862ff1507038edf4c26129abcc11ce8c00bef3
    dangling blob ab0129187ed300114d5d34501add4dc05feaeee0
    dangling commit 5a04331eb72123423888a4fac9922355588f138f
    dangling commit 350da8fc761b2b41d923ccad97cf77e320b0cf34
    dangling commit 8c0e4e66c891f4bffc23c1ee35f4d5eaca2bdb78
    dangling tag 9611c319d012589ea451ce56495f4c9673292615
    .
    . (130 more dangling objects... blobs commits and tags)
    .
    mini-mac7:myRepo $ 
   

and

mini-mac7:itx-mac-agent-broken $ git gc
error: Could not read a2df338e00be26da2f0b84543668ef29b52df5c3
error: Could not read a2df338e00be26da2f0b84543668ef29b52df5c3
error: Could not read a2df338e00be26da2f0b84543668ef29b52df5c3
.
. (40 more times - same object)
.
error: Could not read a2df338e00be26da2f0b84543668ef29b52df5c3
fatal: bad object a2df338e00be26da2f0b84543668ef29b52df5c3
fatal: failed to run repack
mini-mac7:itx-mac-agent-broken $ 

Can anyone please help me revive this repo? First off --- how to fix the 'bad object' issue? I already tried fetching --all from remote, deleting .git/objects/96/11c319d012589ea451ce56495f4c9673292615 directly (only few actually existed, but when rm'ed --- the dangling objects did not disappear.

This repo is important - I already cloned it anew from server, but lots of local stashes and branches there contain work that I need. any attempt to pull/fetch etc. fails with strange errors because of these bad and dangling objects.

Answer 1

A few background points:

1. The failure to push has no relationship to the local problem (whatever the local problem really is).

2. Dangling commits and blobs are normal. The actual problem lies elsewhere (note that actual problems can cause "extra" dangling commits and/or blobs though, so one would not want to just discard all such danglers here: this means don't run git gc on this repository, although in your case fortunately git gc failed and probably did nothing).

Now, the actual problem begins—and perhaps ends—here:

error: Could not read a2df338e00be26da2f0b84543668ef29b52df5c3

From the git fsck output, we know that a2df338e00be26da2f0b84543668ef29b52df5c3 is, or at least was, a commit, before it got damaged:

 broken link from  commit c521a9214898aac031831d97e6a8329342b788c4
               to  commit a2df338e00be26da2f0b84543668ef29b52df5c3

Commit c521a9214898aac031831d97e6a8329342b788c4 itself is OK, but it refers to commit a2df338e00be26da2f0b84543668ef29b52df5c3 by hash ID, and object a2df338e00be26da2f0b84543668ef29b52df5c3 is missing or corrupt.

We also see:

 broken link from  commit 19f55b6374e7adcc42046821c1659bb47df933a8
               to  commit 0d862ff1507038edf4c26129abcc11ce8c00bef3

which suggests that object 0d862ff1507038edf4c26129abcc11ce8c00bef3 is missing as well (perhaps git repack as run by git gc never gets as far as trying to read it so that it does not show up in the git gc output).

This repo is important - I already cloned it anew from server, but lots of local stashes and branches there contain work that I need.

If commit objects a2df338e00be26da2f0b84543668ef29b52df5c3 and 0d862ff1507038edf4c26129abcc11ce8c00bef3 are available in the fresh clone, you can install those objects into the broken clone and see if that makes it work. I have had success with this technique myself in the past. Jump down to the "copying missing objects" section for details.

If those commit objects aren't available, about the best you can probably do is run git fsck --lost-found in the bad repository. This will, for each of the various danglers that Git finds (commits and files that aren't accessible through normal means), write those commits/files into .git/lost-found/. The bad part about this is that these files lose their names. You'll find the files' contents in .git/lost-found/other/<hash-name> (for each hash name); it will be your job to figure out what that file's "useful name" is.

(The files' names may, or may not, be find-able through tree objects, but git fsck doesn't save those tree objects anywhere.)

It's also a very good idea to figure out how the repository got damaged in the first place. The usual case is a power failure or system crash. Other, more alarming cases include disk / SSD failures: if you're getting these, it may be time for new hardware.

Copying missing objects

First, we'll test to see if some object exists in some clone. In my case I'll use ebf3c04b262aa27fbb97f8a0156c2347fecafafb, which I expect to find as a commit:

$ git -C path/to/clone cat-file -t ebf3c04b262aa27fbb97f8a0156c2347fecafafb
commit

If this object exists as a loose object we can just copy it directly to the target Git:

$ cp path/to/clone/.git/objects/eb/f3c04b262aa27fbb97f8a0156c2347fecafafb path/to/original/.git/objects/eb/f3c04b262aa27fbb97f8a0156c2347fecafafb

(using the usual trick that the first two characters of the hash ID become a directory name, and the rest a file within that directory).

In this particular repository, that doesn't work, as ebf3c04b262aa27fbb97f8a0156c2347fecafafb is only in some pack file. There is probably some more clever way to unpack just the one object, but instead of doing that, this is what I would try: git cat-file -p turns the object into a "human readable" copy, and git hash-object -t <type> -w --stdin will turn a human-readable object into a loose object in the .git/objects/ directory and report its ID. So:

$ git -C path/to/good-clone cat-file -p ebf3c04b262aa27fbb97f8a0156c2347fecafafb | git -C path/to/bad-clone hash-object -t commit -w --stdin
ebf3c04b262aa27fbb97f8a0156c2347fecafafb

Having transferred a commit, you might now find a bunch of tree and/or blob objects to be missing, and of course possibly more commits; this process can be repeated, somewhat painfully, one at a time.

There's probably also a simple brute-force method: dump the right pack file(s) into place and run git index-pack, or dump the pack file(s) and index file(s) into place. That's not what I did in the past during one of these kinds of recoveries, though. (I was lucky, I had clobbered just two or three objects and all were loose in the other repo. I had run a bad find ... -remove, failing to skip over the .git directory.)