CODEWITHSUNDEEP
git
Jason R
Jason R

Reputation: 11716

How can I incrementally mirror a Git repository via bundle files?

I have a Git repository on one machine (machine 1) that I would like to keep mirrored on another machine (machine 2). The second machine does not share network access with the first, so simply using git push or git fetch in one direction or another isn't an option. The repository contains many branches, tags, etc. and I would like to replicate that state exactly onto the second machine. The flow of changes is strictly from machine 1 -> machine 2; I don't have to support bringing commits back the other way.

One approach that works is to use Git bundles. That is, if I do the following:

git clone --mirror /path/to/my/source/repo
git bundle create bundle.gitb --all

Then the resulting bundle.gitb file is a perfect mirror of the source repository. I can then copy this to machine 2 and use it as an origin to push changes into the repository mirror.

The main drawback of this approach is that it requires me to transfer the entire repository history each time. In my case, the history is very deep and is several GB in size, so I have to carry all of that data on every update, even if there are just a few new commits.

Therefore, I would like some kind of incremental Git bundle that only contains the commits that I have yet to transfer to machine 2. Is there a good way to do this?

I considered using the --since flag to limit the commits to transfer by commit date, but it's possible that it could cause commits to be missed during the transfer.

I think that in order to meet all of my goals, I would need to be able to provide some kind of manifest of commit SHAs that have already been transferred, and thus I do not want to include in the bundle. So the workflow would be something like:

git clone --mirror /path/to/my/source/repo
# do something here to filter out all commits that have already been transferred
git bundle create bundle.gitb --all

And the resulting file would just contain the incremental commits. Is there a way to achieve this with one of Git's commands?

Upvotes: 1

Views: 734

Answers (1)

fuzzyTew
fuzzyTew

Reputation: 3778

I'm trying this after torek's comment:

#!/usr/bin/env bash
LAST=$(ls | grep \\.bundle$ | sort | tail -n 1)
NEXT=$(date --iso=seconds).bundle
if [ -z "$LAST" ]
then
    git bundle create "$NEXT" --all
else
    git bundle list-heads "$LAST" | cut -d ' ' -f 1 | xargs git bundle create "$NEXT" --all --not
fi

It uses list-heads to output the refs in the last bundle, and xargs with --not to exclude them from the next bundle.

The documentation for --all and --not is in git rev-list --help.

Upvotes: 1

Related Questions