why is git rebase flagging conflicts on new branch specific files

Question

I have a maddening little git rebase issue going on right now.

The brief summary is that there are conflicts on files where there should not possibly be conflicts. I have read many questions on SO and many articles but nothing I have found seems to address an issue quite like this.

What I did

I cloned a remote repository and have been keeping it up to date with git pull --rebase, then rebasing that local master into a local branch:

    git pull --rebase (on master)
    git checkout mybranch
    git rebase master mybranch
    <rebases cleanly>

which looks something like:

    master ----o-o-o-o--------o-o-------------o
                \          \ (pull --rebase)   \(pull --rebase (successful)
    local master ---o-o      o-o-o-o            rebase master to mybranch fails 
                       \            \ (rebase)    
    mybranch           --------A-B   o-A-B-C       

I was only adding new files and this worked great until up until a few weeks ago when I started to get some merge conflicts on files that do not exist on the remote or local master branch.

    git pull --rebase (on master branch and successfully)
    git checkout mybranch
    git rebase master mybranch (conflicts in files that do not exist on master)

I had never merged anything back to to the local or the remote.

There are basically 4 files out of about 100 that I have added that continually get conflicts and during rebase they want to be merged again and again with the same diff over and over and over.

git version 2.9.3

Things I have tried (that did not work or help)

1. Confirmed that I never commited anything on the local master branch and that these conflict files do not exist
2. Cloned another copy of the repo and confirmed I have never commited anything and that these conflict files do not exist
3. I did a "git reset --hard HEAD" before the rebase, no help. removed some local .gitignore files that were not tracked to try to isolate any behavior that may be happening and not consistent between branches.

4. I tried to resolve these conflicts using the mergetool

   I had issues with git losing the conflict resolution with these files since the result of the diff is I always accept the local version that is correct, which git sees no change in.

5. I tried git rebase --skip but this did not seem helpful for the above, so I aborted
6. I tried to power through the merge but gave up after an hour. There are hundreds of commits that I made locally.
7. I turned on rerere.enabled, and tried the above again. rebase attempts to apply the "recorded preimage" but fails and I am asked to merge the non conflicts manually as before.

Things that did work

I can merge into my branch without any problems. The merges are clean and automatic as they should be in this context. This was working quite well for me but I eventually have to push back and cannot propagate this issue.

other information

I merged my branch back into the master, and now I find that doing a git pull --rebase to the master now has

    git checkout mybranch
    git merge master mybranch (success)
    git checkout master
    git merge mybranch master (success)
    git pull --rebase (conflicts in files that dont exist on remote.)

looks something like:

    master ----o-o-o-o--------o-o-------------o-------------------------------------------------------
                \          \ (pull --rebase)   \(pull --rebase (clean)           merged back (clean)  \ rebase
    local master ---o-o      o-o-o-o            o------------------------------o-----------------------X (rebase unclean)
                       \            \ (rebase)   \ merge to mybranch (clean)  /
    mybranch           --------A-B   o-A-B-C      o-----A-B-C-o-...-o-------
                                         (clean, before it stopped working)

My inclination is to give up on this local clone of the repository and merge into a new repository with a filesystem diff however I would really like to understand what is happening and if possible maintain the commit history, but I cannot merge back to the remote, although I imagine it would work, as I believe that would risk propagating this issue (whatever it is).

Answer 1

Based on your comment "[...]the first ~20 commits in the interactive rebase repeat[...]", I believe I can explain the problem.

Suppose the commits are listed like this:

A
B
C
A
B
C

When Git performs the rebase, it attempts to apply each commit one-at-a-time in the order listed.

1. Apply commit A - No issues
2. Apply commit B - No issues
3. Apply commit C - No issues
4. Apply commit A - CONFLICT because these changes have already been applied

Of course, when you manually go to resolve the conflict nothing shows up because the change are identical. This is also why merge works without any problems - it is only comparing the latest versions.

---

So, how do you fix this?

1. git rebase -i master
2. Remove duplicate commit messages. This is tedious, but will only be necessary once
3. git push -f

---

But how did this happen?

This is pure speculation, but is how I got into a similar situation. At some point these commits were part of a branch structure similar to this:

master o-o-o-o-o
            \
branch-1     o-A-B-o
                    \
branch-2             o-o

Then branch-1 was updated via git rebase master:

master o-o-o-o-o
            \   \
branch-1     \   o-A`-B`-o
              \
branch-2       o-A-B-o-o

Then branch-2 was updated via git rebase branch-1:

master o-o-o-o-o
                \
branch-1         o-A`-B`-o
                          \
branch-2                   o-A-B-o-o

And thus the commits became duplicated.

---

What next?

If the duplicated commit messages are commits from master - someone else has re-written the history of master. This is bad form for anyone in a shared repo, and (if possible) you should change the permissions of master to disallow anyone to force push on it.

If the duplicated commit messages are from mybranch, then I'm guessing you had a child-branch at some point and made a mistake when getting the latest changes from master. I use the process described in this article to handle this situation (in short, you need to use the --onto flag when rebasing branch-2).