"Index-pack failed"

Question

upon cloning a git repository, I get an error message that I can't quite grasp.

$ git clone git@pinocchio.unibe.ch:group07
Initialized empty Git repository in /cygdrive/C/Users/Martin Bigler/p2/group07/.
remote: Counting objects: 2269, done.
remote: Compressing objects: 100% (1936/1936), done.
git: 'index-pack' is not a git-command. See 'git --help'.

What could possibly cause this behavior?

Answer 1

Is this similar to ticket 269?

git index-pack isn't a built-in in git.exe, so git.exe needs to find git-index-pack.exe in $GIT_EXEC_PATH (which should usually be "/libexec/git-core/"). Do you have "/libexec/git-core/git-index-pack.exe"?

Because if it is, this is the server which causes the error, not the locally installed git doing the push.

You can try logging interactively and check index-pack is available:

$ ssh git#***.com@***.com
Enter passphrase for key '/c/Users/***/.ssh/id_rsa':
Last login: Tue Feb  9 13:48:32 2010 from ***
-bash-3.2$ git version
git version 1.6.1
-bash-3.2$ git-index-pack
usage: git index-pack [-v] [-o <index-file>] [{ ---keep | --keep=<msg> }] [--strict] 
{ <pack-file> | --stdin [--fix-thin] [<pack-file>] }

That test prompted the following answer:

Your git-index-pack is found when logging in interactively.
But apparently not when you do not log in interactively.
That suggests that you adjusted your PATH appropriately in $HOME/.profile or $HOME/.bash_profile, but not in HOME/.bashrc

And the conclusion:

My solution is:

ssh user@server
cp .bash_profile .bashrc

---

Note that with Git 2.25.2 (March 2020), the index-pack code now diagnoses a bad input packstream that records the same object twice when it is used as delta base; the code used to declare a software bug when encountering such an input, but it is an input error.

See commit a217810 (03 Feb 2020) by Jeff King (peff).
^{(Merged by Junio C Hamano -- gitster -- in commit 7b029eb, 14 Feb 2020)}

index-pack: downgrade twice-resolved REF_DELTA to die()

^{Signed-off-by: Jeff King}

When we're resolving a REF_DELTA, we compare-and-swap its type from REF_DELTA to whatever real type the base object has, as discussed in ab791dd138 ("index-pack: fix race condition with duplicate bases", 2014-08-29, Git v2.2.0-rc0 -- merge).
If the old type wasn't a REF_DELTA, we consider that a BUG(). But as discussed in that commit, we might see this case whenever we try to resolve an object twice, which may happen because we have multiple copies of the base object.

So this isn't a bug at all, but rather a sign that the input pack is broken. And indeed, this case is triggered already in t5309.5 and t5309.6, which create packs with delta cycles and duplicate bases. But we never noticed because those tests are marked expect_failure.

Those tests were added by b2ef3d9ebb ("test index-pack on packs with recoverable delta cycles", 2013-08-23, Git v1.8.5-rc0 -- merge listed in batch #4), which was leaving the door open for cases that we theoretically could handle.
And when we see an already-resolved object like this, in theory we could keep going after confirming that the previously resolved child->real_type matches base->obj->real_type.

But:

• enforcing the "only resolve once" rule here saves us from an infinite loop in other parts of the code.
  If we keep going, then the delta cycle in t5309.5 causes us to loop infinitely, as find_ref_delta_children() doesn't realize which objects have already been resolved.
  So there would be more changes needed to make this case work, and in the meantime we'd be worse off.

• any pack that triggers this is broken anyway.
  It either has a duplicate base object, or it has a cycle which causes us to bring in a duplicate via --fix-thin.
  In either case, we'd end up rejecting the pack in write_idx_file(), which also detects duplicates.

So the tests have little value in documenting what we could be doing (and have been neglected for 6+ years).
Let's switch them to confirming that we handle this case cleanly (and switch out the BUG() for a more informative die() so that we do so).

---

With Git 2.30 (Q1 2021), Processes that access packdata while the .idx file gets removed (e.g. while repacking) did not fail or fall back gracefully as they could.

See commit 506ec2f, commit c8a45eb (25 Nov 2020) by Taylor Blau (ttaylorr).
^{(Merged by Junio C Hamano -- gitster -- in commit 6bac6a1, 08 Dec 2020)}

packfile.c: protect against disappearing indexes

^{Co-authored-by: Jeff King
Signed-off-by: Taylor Blau}

In 17c35c8969 ("packfile: skip loading index if in multi-pack-index", 2018-07-12, Git v2.20.0-rc0 -- merge listed in batch #1) we stopped loading the .idx file for packs that are contained within a multi-pack index.

This saves us the effort of loading an .idx and doing some lightweight validity checks by way of 'packfile.c:load_idx()', but introduces a race between processes that need to load the index (e.g., to generate a reverse index) and processes that can delete the index.

For example, running the following in your shell:

$ git init repo && cd repo
$ git commit --allow-empty -m 'base'
$ git repack -ad && git multi-pack-index write  

followed by:

$ rm -f .git/objects/pack/pack-*.idx
$ git rev-parse HEAD | git cat-file --batch-check='%(objectsize:disk)'  

will result in a segfault prior to this patch.

What's happening here is that we notice that the pack is in the multi-pack index, and so don't check that it still has a .idx.
When we then try and load that index to generate a reverse index, we don't have it, so the call to 'find_pack_revindex()' in 'packfile.c:packed_object_info()' returns NULL, and then dereferencing it causes a segfault.

Of course, we don't ever expect someone to remove the index file by hand, or to be in a state where we never wrote it to begin with (yet find that pack in the multi-pack-index). But, this can happen in a timing race with 'git repack -ad^'(man), which removes all existing packs after writing a new pack containing all of their objects.

Avoid this by reverting the hunk of 17c35c8969 which stops loading the index when the pack is contained in a MIDX.
This makes the latter half of 17c35c8969 useless, since we'll always have a non-NULL 'p->index_data', in which case that if statement isn't guarding anything.

These two together effectively revert 17c35c8969, and avoid the race explained above.

---

With Git 2.31 (Q1 2021), introduce an on-disk file to record revindex for packdata, which traditionally was always created on the fly and only in-core.

See commit 6885cd7 (28 Jan 2021), and commit ec8e776, commit e8c58f8, commit 35a8a35, commit 1615c56, commit c977334, commit e37d0b8, commit 84d5449, commit 8ef50d9, commit 2f4ba2a (25 Jan 2021) by Taylor Blau (ttaylorr).
^{(Merged by Junio C Hamano -- gitster -- in commit 3c12d0b, 12 Feb 2021)}

pack-revindex: ensure that on-disk reverse indexes are given precedence

^{Signed-off-by: Taylor Blau}

When an on-disk reverse index exists, there is no need to generate one in memory.
In fact, doing so can be slow, and require large amounts of the heap.

Let's make sure that we treat the on-disk reverse index with precedence (i.e., that when it exists, we don't bother trying to generate an equivalent one in memory) by teaching Git how to conditionally die() when generating a reverse index in memory.

Then, add a test to ensure that when (a) an on-disk reverse index exists, and (b) when setting GIT_TEST_REV_INDEX_DIE_IN_MEMORY, that we do not die, implying that we read from the on-disk one.

---

With Git 2.34 (Q4 2021), the order in which various files that make up a single (conceptual) packfile has been reevaluated and straightened up.
This matters in correctness, as an incomplete set of files must not be shown to a running Git.

See commit 4bc1fd6, commit 2ec02dd, commit 8737dab, commit 66833f0 (09 Sep 2021), and commit 0c41a88 (08 Sep 2021) by Ævar Arnfjörð Bjarmason (avar).
See commit 522a5c2, commit 4e58ced, commit 16a8690, commit ae44b5a (09 Sep 2021) by Taylor Blau (ttaylorr).
^{(Merged by Junio C Hamano -- gitster -- in commit a1af533, 20 Sep 2021)}

index-pack: refactor renaming in final()

^{Signed-off-by: Ævar Arnfjörð Bjarmason
Signed-off-by: Taylor Blau}

Refactor the renaming in final() into a helper function, this is similar in spirit to a preceding refactoring of finish_tmp_packfile() in pack-write.c.

Before e37d0b8 ("builtin/index-pack.c: write reverse indexes", 2021-01-25, Git v2.31.0-rc0 -- merge listed in batch #8) it probably wasn't worth it to have this sort of helper, due to the differing "else if" case for "pack" files v.s.
"idx" files.

But since we've got "rev" as well now, let's do the renaming via a helper, this is both a net decrease in lines, and improves the readability, since we can easily see at a glance that the logic for writing these three types of files is exactly the same, aside from the obviously differing cases of "*final_name" being NULL, and "make_read_only_if_same" being different.

---

Also, Geometric repacking ("git repack --geometric=<n>"^(man)) in a repository that borrows from an alternate object database had various corner case bugs, which have been corrected with Git 2.41 (Q2 2023).

See commit d85cd18, commit 932c16c, commit 19a3a7b, commit f302841, commit 752b465, commit 732194b, commit b7b8f04, commit 5186134, commit 3d74a23, commit ceb96a1 (14 Apr 2023) by Patrick Steinhardt (pks-t).
^{(Merged by Junio C Hamano -- gitster -- in commit 36628c5, 25 Apr 2023)}

pack-objects: fix error when same packfile is included and excluded

^{Signed-off-by: Patrick Steinhardt}

When passing the same packfile both as included and excluded via the --stdin-packs option, then we will return an error because the excluded packfile cannot be found.
This is because we will only set the util pointer for the included packfile list if it was found, so that we later die when we notice that it's in fact not set for the excluded packfile list.

Fix this bug by always setting the util pointer for both the included and excluded list entries.

---

And, still with Git 2.41 (Q2 2023): the on-disk reverse index that allows mapping from the pack offset to the object name for the object stored at the offset has been enabled by default.

See commit 9f7f10a, commit a8dd7e0, commit dbcf611, commit 2a250d6, commit 65308ad, commit b77919e, commit 3969e6c (12 Apr 2023) by Taylor Blau (ttaylorr).
^{(Merged by Junio C Hamano -- gitster -- in commit 849c8b3, 27 Apr 2023)}

pack-revindex: introduce GIT_TEST_REV_INDEX_DIE_ON_DISK

^{Signed-off-by: Taylor Blau
Acked-by: Derrick Stolee}

In ec8e776 ("pack-revindex: ensure that on-disk reverse indexes are given precedence", 2021-01-25, Git v2.31.0-rc0 -- merge listed in batch #8), we introduced GIT_TEST_REV_INDEX_DIE_IN_MEMORY to abort the process when Git generated a reverse index from scratch.

ec8e776 was about ensuring that Git prefers a .rev file when available over generating the same information in memory from scratch.

In a subsequent patch, we'll introduce pack.readReverseIndex, which may be used to disable reading ".rev" files when available.
In order to ensure that those files are indeed being ignored, introduce an analogous option to abort the process when Git reads a ".rev" file from disk.

---

With Git 2.44 (Q1 2024), batch 15, the "--fsck-objects" option of "git index-pack"^(man) now can take the optional parameter to tweak the severity of different fsck errors.

See commit 0f8edf7, commit 2811019 (01 Feb 2024) by John Cai (john-cai).
^{(Merged by Junio C Hamano -- gitster -- in commit 2c90347, 08 Feb 2024)}

index-pack: --fsck-objects to take an optional argument for fsck msgs

^{Reviewed-by: Christian Couder
Signed-off-by: John Cai}

git-index-pack^(man) has a --strict option that can take an optional argument to provide a list of fsck issues to change their severity.
--fsck-objects does not have such a utility, which would be useful if one would like to be more lenient or strict on data integrity in a repository.

Like --strict, allow --fsck-objects to also take a list of fsck msgs to change the severity.

Remove the "For internal use only" note for --fsck-objects, and document the option.
This won't often be used by the normal end user, but it turns out it is useful for Git forges like GitLab.

git index-pack now includes in its man page:

--fsck-objects[=<msg-id>=<severity>...]

Die if the pack contains broken objects, but unlike --strict, don't choke on broken links. If the pack contains a tree pointing to a .gitmodules blob that does not exist, prints the hash of that blob (for the caller to check) after the hash that goes into the name of the pack/idx file (see "Notes").

An optional comma-separated list of <msg-id>=<severity> can be passed to change the severity of some possible issues, e.g., --fsck-objects="missingEmail=ignore,badTagName=ignore".
See the entry for the fsck.<msg-id> configuration options in git fsck for more information on the possible values of <msg-id> and <severity>.