Is it possible to rebase branches along with their unmerged child branches in git?

Question

I have a project and its repository with the following branching model:

• master
• ↳ dev
•   ↳ feature-####
•     ↳ subtask-####-####

Because of pull requests, every commit is encouraged to be merged to its parent branch without fast-forwarding (done via the web-interface)

• subtask-####-#### ⮕ feature-####
• feature-#### ⮕ dev
• dev ⮕ master

For example,

  * merge feature-2 into dev
  |\
  | * merge subtask 2-1 into feature-2
  | |\
  | | * subtask 2-1
  | |/
  | * feature-2
  |/
  |
  |   * subtask 1-3 (the subtask is not done yet)
  |  /
  | * merge subtask-1-2 into feature-1
  | |\
  | | * subtask 1-2
  | | * subtask 1-2
  | |/
  | * merge subtask-1-1 into feature-1
  | |\
  | | * subtask-1-1
  | |/
  | * feature-1
  |/
  * dev
 /
* master
.
.
.

Now suppose I want to rebase the not-yet-completed feature-1 branch onto the dev branch where the feature-2 branch is already merged to (it's considered safe because the feature branches are supposed not to change the same code). The way I see it is:

git checkout feature-1
git rebase -p dev # now the feature-1 branch is on-top of the dev branch preserving the merges
git checkout subtask-1-3
git rebase -p feature-1

But the last command fails with the following output:

error: commit cd801c0b02c9a2a27c58ab6e3245bf526099f12c is a merge but no -m option was given.
fatal: cherry-pick failed
Could not pick cd801c0b02c9a2a27c58ab6e3245bf526099f12c

As far as I understand, rebase uses cherry-pick under the hood and the latter requires the -m flag, and this flag is not passed with rebase. I'm not sure, but simple git rebase --continue seems to be a work around it and the history seems to be kept according to the branching model. git rebase --continue might be required to be executed a few times until the rebase completes.

My questions are:

• Is it safe to rebase the history this way to avoid merge-tangled history?
• How do I tell git not to stop for cherry-picking merge commits during rebase, and are my assumptions regarding git rebase --continue correct?
• Or, if it's ever possible, how do I rebase the whole feature branch along with its all unmerged child branches onto a new commit of its parent branch? For example, I would like to rebase the whole feature-1 with subtask-1-3 automatically. I understand that git does not have a this-is-a-parent/child-branch concept, but any way to specify the relations between the branches would be perfectly fine.

My git version is git version 2.15.1. Thank you.

Answer 1

I've implemented a Python script the can implement this. The script is NOT well-tested, but it seems to work fine for non-conflict cases.

The concept behind the script is relations between branches. A directed graph is generated from the relations in order to calculate the rebase order. The relations are stored in ./.git/xrebase-topology. However, there some potentially dangerous or even destructive cases:

• If there are conflicts, the script aborts suggesting to resolve conflicts, proceed with merge or rebase and then re-run the script.
• The script does not seem to work fine with empty commits.

Install:

ln -s "$(pwd)/git-xrebase" "$(git --exec-path)/git-xrebase"

Uninstall:

rm "$(git --exec-path)/git-xrebase"

For the repo I asked for the associations are generated with:

git xrebase add master dev
git xrebase add dev feature-1
git xrebase add feature-1 subtask-1-1 subtask-1-2 subtask-1-3

Then the dev branch and its descendants can be rebased by a single command:

git xrebase rebase dev

Example output for the perfect case:

master<-dev: checkout to dev
master<-dev: rebase...
dev<-feature-1: checkout to feature-1
dev<-feature-1: rebase...
feature-1<-subtask-1-1: checkout to subtask-1-1
feature-1<-subtask-1-1: rebase...
feature-1<-subtask-1-2: checkout to subtask-1-2
feature-1<-subtask-1-2: rebase...
feature-1<-subtask-1-3: checkout to subtask-1-3
feature-1<-subtask-1-3: rebase...
done
master<-dev: checkout to dev
master<-dev: rebase...
dev<-feature-1: checkout to feature-1
dev<-feature-1: rebase...
feature-1<-subtask-1-1: checkout to subtask-1-1
feature-1<-subtask-1-1: rebase...
feature-1<-subtask-1-2: checkout to subtask-1-2
feature-1<-subtask-1-2: rebase...
done
master<-dev: checkout to dev
master<-dev: rebase...
dev<-feature-1: checkout to feature-1
dev<-feature-1: rebase...
feature-1<-subtask-1-1: checkout to subtask-1-1
feature-1<-subtask-1-1: rebase...
done
master<-dev: checkout to dev
master<-dev: rebase...
dev<-feature-1: checkout to feature-1
dev<-feature-1: rebase...
done

git-xrebase.py

#!/usr/bin/env python

import collections
import git
import itertools
import os
import re
import sys

__repo = git.Repo(search_parent_directories = True)
__git = git.cmd.Git(__repo.working_tree_dir)
__topology_file_path = os.path.join(__repo.working_tree_dir, '.git', 'xrebase-topology')

class __XRebaseException(Exception):
    pass

def __peek(callback, sequence):
    for e in sequence:
        callback(e)
        yield e

def __read_file_lines(path, ignore_no_file = False):
    if not ignore_no_file and not os.path.isfile(path):
        return
        yield
    l = len(os.linesep)
    for line in open(path, 'r'):
        yield line[:-l] if line.endswith(os.linesep) else line

def __write_file_lines(path, lines):
    with open(path, 'w') as file:
        for line in lines:
            file.write(line)
            file.write(os.linesep)

class ParentAndChild:

    def __init__(self, parent, child):
        self.parent = parent
        self.child = child

    def __str__(self):
        return '(%s<-%s)' % (self.parent, self.child)

    def __hash__(self):
        return hash((self.parent, self.child))

    def __eq__(self, other):
        if other == None:
            return False
        return self.parent == other.parent and self.child == other.child

def __compare_parent_child(pc1, pc2):
    parent_cmp = cmp(pc1.parent, pc2.parent)
    if parent_cmp != 0:
        return parent_cmp
    child_cmp = cmp(pc1.child, pc2.child)
    return child_cmp

def __read_raw_topology():
    whitespace_re = re.compile('\s*')
    for line in __read_file_lines(__topology_file_path):
        if len(line) > 0:
            split = whitespace_re.split(line.strip())
            if len(split) != 2:
                raise __XRebaseException('syntax error: %s' % line)
            [parent, child] = split
            yield ParentAndChild(parent, child)

def __write_raw_topology(raw_topology):
    sorted_raw_topology = sorted(set(raw_topology), cmp = __compare_parent_child)
    def lines():
        for parent_and_child in sorted_raw_topology:
            yield '%s %s' % (parent_and_child.parent, parent_and_child.child)
    __write_file_lines(__topology_file_path, lines())

class Node:

    def __init__(self, name):
        self.name = name
        self.parent = None
        self.children = collections.OrderedDict()

    def __hash__(self):
        return hash((self.name))

    def __eq__(self, other):
        if other == None:
            return False
        return self.name == other.name

    def __str__(self):
        return '(%s->%s->[%s])' % (self.name, self.parent.name if self.parent != None else '?', ','.join(map(lambda node: node.name, self.children.values())))

def __build_graph_index(raw_topology):
    graph_index = {}
    def get_node(name):
        if not (name in graph_index):
            node = Node(name)
            graph_index[name] = node
            return node
        return graph_index[name]
    for parent_and_child in raw_topology:
        parent_node = get_node(parent_and_child.parent)
        child_node = get_node(parent_and_child.child)
        parent_node.children[parent_and_child.child] = child_node
        child_node.parent = parent_node
    return graph_index

def __find_graph_index_roots(nodes):
    for node in nodes:
        if node.parent == None:
            yield node

def __dfs(nodes, get_children, consume, level = 0):
    for node in nodes:
        consume(node, level)
        __dfs(get_children(node).values(), get_children, consume, level + 1)

def __dfs_1_go(nodes, get_children, consume, level = 0, visited_nodes = list()):
    for node in nodes:
        if node in visited_nodes:
            raise __XRebaseException('%s causes infinite recursion' % node);
        consume(node, level)
        visited_nodes.append(node);
        __dfs_1_go(get_children(node).values(), get_children, consume, level + 1, visited_nodes)

def __do_add(parent, children):
    new_parent_and_children = list(map(lambda child: ParentAndChild(parent, child), children))
    def check(old_parent_and_child):
        if old_parent_and_child in new_parent_and_children:
            print '%s already exists' % old_parent_and_child
    raw_topology = itertools.chain(__peek(check, __read_raw_topology()), new_parent_and_children)
    __write_raw_topology(raw_topology)

def __do_clear():
    if os.path.isfile(__topology_file_path):
        os.remove(__topology_file_path)
    else:
        raise __XRebaseException('cannot clear: %s does not exist' % __topology_file_path)

def __do_help():
    print '''available commands:
    add <parent_branch> [child_branch...]
        add parent/child branch associations
    clear
        clear all parent/child branch associations
    help
        show this help
    list
        show branches list
    rebase [branch...]
        rebase branches
    remove <parent_branch> [child_branch...]
        remove parent/child branch associations
    tree
        show branches in a tree'''

def __do_list():
    for parent_and_child in __read_raw_topology():
        print parent_and_child

def __do_rebase(branches):
    if __repo.is_dirty():
        raise __XRebaseException('cannot rebase: dirty')
    graph_index = __build_graph_index(__read_raw_topology())
    nodes_to_rebase = []
    for branch in branches:
        if not (branch in graph_index):
            raise __XRebaseException('cannot found %s in %s' % (branch, graph_index.keys()))
        nodes_to_rebase.append(graph_index[branch])
    ordered_nodes_to_rebase = []
    __dfs_1_go(nodes_to_rebase, lambda node: node.children, lambda node, level: ordered_nodes_to_rebase.append(node))
    for node in ordered_nodes_to_rebase:
        if not node.name in __repo.branches:
            raise __XRebaseException('%s does not exist. deleted?' % node.name)
    original_refs = {}
    for node in ordered_nodes_to_rebase:
        original_refs[node.name] = __repo.branches[node.name].object.hexsha
    original_branch = __repo.head.ref
    success = True
    try:
        stdout_re = re.compile('^stdout: \'(.*)\'$', re.DOTALL)
        stderr_re = re.compile('^stderr: \'(.*)\'$', re.DOTALL)
        for node in filter(lambda node: node.parent, ordered_nodes_to_rebase):
            line_prefix = '%s<-%s' % (node.parent.name, node.name)
            def fix_cherry_pick():
                while True:
                    try:
                        print '%s: cherry-pick failed. trying to proceed with rebase --continue...' % line_prefix
                        __git.rebase('--continue')
                    except git.exc.GitCommandError as cherry_pick_ex:
                        cherry_pick_message_match = stdout_re.search(cherry_pick_ex.stdout.strip())
                        cherry_pick_message = (cherry_pick_message_match.group(1) if cherry_pick_message_match else '')
                        cherry_pick_error_message_match = stderr_re.search(cherry_pick_ex.stderr.strip())
                        cherry_pick_error_message = cherry_pick_error_message_match.group(1) if cherry_pick_error_message_match else ''
                        if cherry_pick_error_message.startswith('Could not pick '):
                            continue
                        elif cherry_pick_error_message == 'No rebase in progress?':
                            print '%s: cherry-pick fixed' % line_prefix
                            return True
                        elif  cherry_pick_message.find('You must edit all merge conflicts') != -1:
                            print 'please fix the conflicts and then re-run: %s' % ('git xrebase rebase %s' % ' '.join(branches))
                            return False
                        else:
                            raise __XRebaseException('cannot fix cherry-pick: %s' % str(cherry_pick_ex))
            print '%s: checkout to %s' % (line_prefix, node.name)
            __repo.branches[node.name].checkout()
            try:
                print '%s: rebase...' % (line_prefix)
                __git.rebase('-p', node.parent.name)
            except git.exc.GitCommandError as rebase_ex:
                rebase_error_message_match = stderr_re.search(rebase_ex.stderr.strip())
                rebase_error_message = rebase_error_message_match.group(1) if rebase_error_message_match else ''
                if rebase_error_message.startswith('Could not pick '):
                    if not fix_cherry_pick():
                        success = False
                        break
                elif rebase_error_message == 'Nothing to do':
                    print '%s: done' % line_prefix
                    continue
                else:
                    raise __XRebaseException('cannot rebase: %s' % rebase_error_message)
        print 'done' if success else 'could not rebase'
    except Exception as ex:
        if isinstance(ex, git.exc.GitCommandError):
            sys.stderr.write('git: %s\n' % ex.stderr.strip())
        try:
            __git.rebase('--abort')
        except git.exc.GitCommandError as git_ex:
            sys.stderr.write('git: %s\n' % git_ex.stderr.strip())
        for (branch, hexsha) in original_refs.iteritems():
            print 'recovering %s back to %s' % (branch, hexsha)
            __repo.branches[branch].checkout()
            __repo.head.reset(commit = hexsha, index = True, working_tree = True)
        raise __XRebaseException(str(ex))
    finally:
        if success:
            original_branch.checkout()

def __do_remove(parent, children):
    raw_topology = list(__read_raw_topology())
    for parent_and_child in map(lambda child: ParentAndChild(parent, child), children):
        if not (parent_and_child in raw_topology):
            print '%s not found' % parent_and_child
        else:
            raw_topology.remove(parent_and_child)
    __write_raw_topology(raw_topology)

def __do_tree():
    graph_index = __build_graph_index(__read_raw_topology())
    roots = __find_graph_index_roots(graph_index.values())
    def __print(node, level = 0):
        print '%s%s' % (' ' * level, node.name)
    __dfs(roots, lambda node: node.children, __print)

# entry point

def __dispatch(command, command_args):
    if command == 'add':
        __do_add(command_args[0], command_args[1:])
    elif command == 'clear':
        __do_clear()
    elif command == 'help':
        __do_help()
    elif command == 'list':
        __do_list()
    elif command == 'rebase':
        __do_rebase(command_args[0:])
    elif command == 'remove':
        __do_remove(command_args[0], command_args[1:])
    elif command == 'tree':
        __do_tree()
    else:
        raise __XRebaseException('unrecognized command: %s' % command)

if __name__ == '__main__':
    command = sys.argv[1] if len(sys.argv) > 1 else 'help'
    command_args = sys.argv[2:]
    try:
        __dispatch(command, command_args)
    except __XRebaseException as ex:
        sys.stderr.write('fatal: %s\n' % ex.message)

I believe I missed some important things, and it would be nice to have a similar feature in git.