How to merge git repositories in subfolders without losing branches and tags?

Question

While this question would seem to be a duplicate or many previous ones (and tens of blog posts dealing with similar subject), I need to make you aware that I am looking for an approach that would allow me to script it.

• all branches and tags are supposed to merged
• branches and tags are supposed to be prefixed in order to avoid overlapping
• files would not overlap because we import to a new subfolder.
• Incremental merge process: be able to merge new repositories later as I plan to migrate to a monorepo setup slowly, one repository at a time.

The script I am trying to build should work like this:

./merge-repo monorepo/subdir/foo git@github.com:example/foo.git

This would assume that the target repository (monorepo) is already cloned inside monorepo/ folder and that it does not have any local changes.

For example if the foo.git would have two branches master and develop and one tag names v1.0, after performing the merge, I would expect to see branches foo-master and foo-develop and the tag foo-v1.0 inside monorepo repository.

I read tens of articles (or scripts) but so far I was not able to find one that explains how can I obtain this.

Clarifications

• If I do have a README.txt inside the root of the foo.git repo, I would expect to see this file inside monorepo/subdir/foo/README.txt. That's the only approach that would allow me to merge repositories without conflicts, if these are going to become subdirectories of the big repo (subdirectories that did not exist before).

Update

While I am not ready yet to say that the issue is solved, I ended up writing this bash script that is supposed to perform the merge after patching the source repository. Have a look at it https://github.com/ssbarnea/monorepo/blob/master/git-monorepo-add.sh

PS. I will be back with a proper answer as soon I do have a reliable solution, tested with many repositories.

Answer 1

Basically, you need to do two things:

1. Import all the underlying git objects.
2. Merge user-facing structure of the repositories (branches, tags, etc.)

You do the former by simply fetching all the branches and tags of the remote repository. Since the objects are indexed by hashes, taking the union is a trivial task, you have to do nothing.

At this point you have all the necessary objects inside your repository. You’ve got all the remote branches, and you can list them e.g. by running git branch -r --list '<remote_name>/*' (actually, you don’t have to do anything if you are ok with this name pattern).

Tags, as far as I can tell, are a little bit trickier. They are not scoped by remotes, so you can’t just fetch them all together and rename later. This means, you’ll have to manually obtain the list of tags (git ls-remote --tags <remote_name>), iterate over them creating the ones with altered names and fetch.

Answer 2

You can pull a remote repositories branches and tags into namespaced (i.e., <prefix>) refs with something like:

git fetch --no-tags <url> \
    refs/heads/*:refs/heads/<prefix>/* \
    refs/tags/*:refs/tags/<prefix>/*

With your example, this would give you branches foo/master and foo/develop and a tag named foo/v1.0. Not exactly what you asked for, but maybe that's fine. Note that --no-tags is specified to avoid fetching tags into the "global" namespace.

At this point you probably want to merge one of these branches into you local repo's master. Here is one way to go about this:

$ git merge -s ours --no-commit foo/master
$ git read-tree --prefix=foo/ -u foo/master
$ git commit -m "merged in `master' from project foo"