Git gc uses a lot of memory, even I limited it

Question

I am running git on a shared host (uberspace.de) and hit the memory limit when doing git push or git gc. Memory consumption reported by them is around 1500 MB.

So I followed the advice in

https://stackoverflow.com/a/8761464/288568

It seemed to work once, but after lots of commits it does not help anymore and the memory consumption is again around 1500MB and the git pack process gets killed.

I commit a 120 MB SQL dump with very fews changes (around 5 lines each time) - so the packed repository is around 150 MB only, but unpacked around 3 GB (and I hit bitbucket's limit).

Which pmap I see it is opening a lot of pack objects - is it that which adds up to the high memory? How can this be limited?

Finally I get "pack-objects died of signal 15" and a mail from the hoster, that my process was killed.

sh-4.1$ git config --list|grep pack
pack.windowmemory=25m
pack.packsizelimit=25m
pack.threads=1
pack.deltecachesize=25m
sh-4.1$ git gc
Counting objects: 2586, done.
^Zmpressing objects:  15% (163/1085)   
[1]+  Stopped(SIGTSTP)        git gc
sh-4.1$ bg
[1] git gc &
sh-4.1$ pmap 14190
14190:   git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago --local --delta-base-offset .git/objects/pack/.tmp-14187-pack
0000000000400000   1848K r-x--  /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007ce000     36K rw---  /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007d7000    280K rw---    [ anon ]
0000000000da1000   1680K rw---    [ anon ]
00007f735dc5d000 161648K rw---    [ anon ]
00007f7377555000 149580K rw---    [ anon ]
00007f73915ee000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.pack
00007f73990b6000 125592K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.pack
00007f73a0b5c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.pack
00007f73a8624000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.pack
00007f73b00ec000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.pack
00007f73b7bb4000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.pack
00007f73bf67c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.pack
00007f73c7144000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.pack
00007f73cec0c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.pack
00007f73d66d4000 125348K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.pack
00007f73de13d000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.pack
00007f73e5c05000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.pack
00007f73ed6cb000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.pack
00007f73f5191000 120872K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.pack
00007f73fc79b000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.pack
00007f7404263000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.pack
00007f740bd29000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.pack
00007f74137ef000 120868K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.pack
00007f741adf8000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.pack
00007f74228c0000    484K rw---    [ anon ]
00007f7422939000  96840K r----  /usr/lib/locale/locale-archive
00007f74287cb000      8K r-x--  /lib64/libdl-2.12.so
00007f74287cd000   2048K -----  /lib64/libdl-2.12.so
00007f74289cd000      4K r----  /lib64/libdl-2.12.so
00007f74289ce000      4K rw---  /lib64/libdl-2.12.so
00007f74289cf000   1576K r-x--  /lib64/libc-2.12.so
00007f7428b59000   2048K -----  /lib64/libc-2.12.so
00007f7428d59000     16K r----  /lib64/libc-2.12.so
00007f7428d5d000      8K rw---  /lib64/libc-2.12.so
00007f7428d5f000     16K rw---    [ anon ]
00007f7428d63000     92K r-x--  /lib64/libpthread-2.12.so
00007f7428d7a000   2048K -----  /lib64/libpthread-2.12.so
00007f7428f7a000      4K r----  /lib64/libpthread-2.12.so
00007f7428f7b000      4K rw---  /lib64/libpthread-2.12.so
00007f7428f7c000     16K rw---    [ anon ]
00007f7428f80000     28K r-x--  /lib64/librt-2.12.so
00007f7428f87000   2044K -----  /lib64/librt-2.12.so
00007f7429186000      4K r----  /lib64/librt-2.12.so
00007f7429187000      4K rw---  /lib64/librt-2.12.so
00007f7429188000   1768K r-x--  /usr/lib64/libcrypto.so.1.0.1e
00007f7429342000   2044K -----  /usr/lib64/libcrypto.so.1.0.1e
00007f7429541000    108K r----  /usr/lib64/libcrypto.so.1.0.1e
00007f742955c000     48K rw---  /usr/lib64/libcrypto.so.1.0.1e
00007f7429568000     16K rw---    [ anon ]
00007f742956c000     84K r-x--  /lib64/libz.so.1.2.3
00007f7429581000   2044K -----  /lib64/libz.so.1.2.3
00007f7429780000      4K r----  /lib64/libz.so.1.2.3
00007f7429781000      4K rw---  /lib64/libz.so.1.2.3
00007f7429782000    128K r-x--  /lib64/ld-2.12.so
00007f74297a6000    464K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.pack
00007f742981a000     52K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.pack
00007f7429827000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.pack
00007f7429828000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.pack
00007f7429829000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.pack
00007f742982a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.pack
00007f742982b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.pack
00007f742982c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.pack
00007f742982d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.pack
00007f742982e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.pack
00007f742982f000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.pack
00007f7429831000     20K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.pack
00007f7429836000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.pack
00007f7429838000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.pack
00007f742983a000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.pack
00007f742983c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.pack
00007f742983d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.pack
00007f742983e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.pack
00007f742983f000     20K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.pack
00007f7429875000      4K rw---    [ anon ]
00007f7429876000    400K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.pack
00007f74298da000    512K rw---    [ anon ]
00007f742995a000     44K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.idx
00007f7429965000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.idx
00007f7429966000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.idx
00007f7429967000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.idx
00007f7429968000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.idx
00007f7429969000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.idx
00007f742996a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.idx
00007f742996b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.idx
00007f742996c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.idx
00007f742996d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.idx
00007f742996e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.idx
00007f742996f000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.idx
00007f7429970000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.idx
00007f7429971000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.idx
00007f7429972000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.idx
00007f7429973000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.idx
00007f7429974000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.idx
00007f7429975000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.idx
00007f7429976000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.idx
00007f7429977000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.idx
00007f7429978000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.idx
00007f7429979000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.idx
00007f742997a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.idx
00007f742997b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.idx
00007f742997c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.idx
00007f742997d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.idx
00007f742997e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.idx
00007f742997f000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.idx
00007f7429980000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.idx
00007f7429981000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.idx
00007f7429982000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.idx
00007f7429983000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.idx
00007f7429984000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.idx
00007f7429985000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.idx
00007f7429986000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.idx
00007f7429987000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.idx
00007f7429988000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.idx
00007f7429989000     24K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.idx
00007f742998f000     20K rw---    [ anon ]
00007f7429998000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.idx
00007f7429999000     28K r--s-  /usr/lib64/gconv/gconv-modules.cache
00007f74299a0000      4K rw---    [ anon ]
00007f74299a1000      4K r----  /lib64/ld-2.12.so
00007f74299a2000      4K rw---  /lib64/ld-2.12.so
00007f74299a3000      4K rw---    [ anon ]
00007ffe76939000     84K rw---    [ stack ]
00007ffe7698b000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]
 total          2809104K
sh-4.1$ error: pack-objects died of signal 15
error: failed to run repack

Answer 1

Git 2.18 (Q2 2018) should help detect and alleviate excessive memory consumption.

"git gc" in a large repository takes a lot of time as it considers to repack all objects into one pack by default.
The command has been taught to pretend as if the largest existing packfile is marked with ".keep" so that it is left untouched while objects in other packs and loose ones are repacked.

See commit 5af0504, commit 9806f5a, commit 8fc6776, commit 55dfe13, commit ae4e89e, commit ed7e5fc, commit e9e33ab (15 Apr 2018) by Nguyễn Thái Ngọc Duy (pclouds).
^{(Merged by Junio C Hamano -- gitster -- in commit 30b015b, 23 May 2018)}

gc --auto: exclude base pack if not enough mem to "repack -ad"

pack-objects could be a big memory hog especially on large repos, everybody knows that.
The suggestion to stick a .keep file on the giant base pack to avoid this problem is also known for a long time.

Recent patches add an option to do just this, but it has to be either configured or activated manually.
This patch lets git gc --auto activate this mode automatically when it thinks repack -ad will use a lot of memory and start affecting the system due to swapping or flushing OS cache.

gc --auto decides to do this based on an estimation of pack-objects memory usage, which is quite accurate at least for the heap part, and whether that fits in half of system memory (the assumption here is for desktop environment where there are many other applications running).

This mechanism only kicks in if gc.bigBasePackThreshold is not configured. If it is, it is assumed that the user already knows what they want.

The git gc man page now states:

If the number of packs exceeds the value of gc.autoPackLimit, then existing packs (except those marked with a .keep file or over gc.bigPackThreshold limit) are consolidated into a single pack by using the -A option of 'git repack'.

If the amount of memory is estimated not enough for git repack to run smoothly and gc.bigPackThreshold is not set, the largest pack will also be excluded (this is the equivalent of running git gc with --keep-base-pack).
Setting gc.autoPackLimit to 0 disables automatic consolidation of packs.

---

Git 2.30 (Q1 2021) fixes an option name in "gc" documentation.

See commit 793c146, commit db5368b (20 Nov 2020) by Ævar Arnfjörð Bjarmason (avar).
^{(Merged by Junio C Hamano -- gitster -- in commit 1c04cdd, 30 Nov 2020)}

gc docs: change --keep-base-pack to --keep-largest-pack

^{Reported-by: Luc Van Oostenryck
Signed-off-by: Ævar Arnfjörð Bjarmason}

The --keep-base-pack option never existed in git.git. It was the name for the --keep-largest-pack option in earlier revisions of that series before it landed as ae4e89e549 ("gc: add^(man) --keep-largest-pack option", 2018-04-15, Git v2.18.0-rc0 -- merge listed in batch #6).

The later patches in that series weren't changed to also refer to --keep-largest-pack, so we've had this reference to a nonexisting option ever since the feature initially landed.

Those commits were:

1. 55dfe13df9 ("gc: add gc.bigPackThreshold config", 2018-04-15, Git v2.18.0-rc0 -- merge listed in batch #6)
2. 9806f5a7bf ("gc --auto: exclude base pack if not enough mem to "repack -ad"", 2018-04-15, Git v2.18.0-rc0)

Answer 2

maybe solution

I observed, that when I choose the packSizeLimit very small, this is contra productive - it increased the .git directory instead of decreasing it, and this might cause also the bit bucket size limit to be reached (unverified)

1) rsync the repo to another computer with more RAM and pack it properly

2) rsync it back to server with the memory limit

3) increase to pack.packSizeLimit=300m

4) Now after a git gc the repo does not grow like crazy

5) commit the database dump

6) repo doubles its size

7) directly do a git gc

8) size becomes fine

9) now push

let's see how it goes after some days ...

Answer 3

I have some big repository which I have to compress and I did not notice any memrory related options help. If git-pack-objects has to repack some objects, it will have to map roughly same big memory. The only way to manage it is to limit number of objects it has to repack.

In your case object already come compressed in small pack files (one for each push). So the git gc has to repack those small pack files into big ones. You could try to create empty pack-<HASH>.keep files next to the big pack files which were creates by last succesful repack. Then maybe next git gc will not try to deltify those objects again.