git pull yields "error: Updating the following directories would lose untracked files in it" on specific repository

Question

When I use git pull I get the "error: Updating the following directories would lose untracked files in it:" message.

I have two different repositories, which I'll call "good" and "abc". I'm trying to merge them into a parent repository following the procedure described here, in order to keep the original histories. This involves re-writing history with the help of git filter-branch, then pulling both repositories into a newly initialized parent repository. This is the failing step:

git pull abc/

remote: Counting objects: 237, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 237 (delta 129), reused 161 (delta 129)
Receiving objects: 100% (237/237), 54.99 KiB | 0 bytes/s, done.
Resolving deltas: 100% (129/129), done.
From abc
 * branch            HEAD       -> FETCH_HEAD
error: Updating the following directories would lose untracked files in it:
    abc/abc

Now, if I pull the "good" repository it works fine even after re-writing history. Or if I git pull abc before re-writing history, it also goes well. This seems to be specific to the "abc" repo combined with the filter-branch procedure. So the question is, what could possibly be wrong with this repository to cause this error? How do I go about troubleshooting it?

The full sequence of commands and output can be seen below:

:~$ mkdir gitexp
:~$ cd gitexp/
:~/gitexp$ git clone https://github.com/nagev/good

Cloning into 'good'...
remote: Counting objects: 770, done.
remote: Total 770 (delta 0), reused 0 (delta 0), pack-reused 770
Receiving objects: 100% (770/770), 3.84 MiB | 898.00 KiB/s, done.
Resolving deltas: 100% (301/301), done.

:~/gitexp$ git clone https://github.com/nagev/abc
Cloning into 'abc'...
remote: Counting objects: 199, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 199 (delta 129), reused 199 (delta 129), pack-reused 0
Receiving objects: 100% (199/199), 53.52 KiB | 0 bytes/s, done.
Resolving deltas: 100% (129/129), done.

:~/gitexp$ ls
good  abc
:~/gitexp$ cd good/

:~/gitexp/good$ git filter-branch --index-filter 'git ls-files -s | sed "s-\t-&good/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Rewrite 34f94b29c73edbb003f969136dba325a55df052d (43/60) (1 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/master' was rewritten

:~/gitexp/good$ cd ../abc/
:~/gitexp/abc$ git filter-branch --index-filter 'git ls-files -s | sed "s-\t-&abc/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Rewrite e7ede9c143daa3d5188a4632a327dda75b1a4eaf (22/38) (1 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/master' was rewritten

:~/gitexp/abc$ cd ..
:~/gitexp$ git init
Initialized empty Git repository in /home/nagev/gitexp/.git/

:~/gitexp$ git pull abc/
remote: Counting objects: 237, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 237 (delta 129), reused 161 (delta 129)
Receiving objects: 100% (237/237), 54.99 KiB | 0 bytes/s, done.
Resolving deltas: 100% (129/129), done.
From abc
 * branch            HEAD       -> FETCH_HEAD
error: Updating the following directories would lose untracked files in it:
    abc/abc

Aborting
:~/gitexp$ cd abc/
:~/gitexp/abc$ tree
.
└── abc
    ├── vrct
    ├── jlsg.capnp
    ├── nabc
    │   ├── forest.py
    │   ├── info.py
    │   ├── datalog.py
    │   ├── rbsd.py
    │   ├── rotation.py
    │   ├── text.py
    │   ├── __init__.py
    │   ├── master.py
    │   ├── structure.py
    │   ├── calculate.py
    │   └── date.py
    ├── abc
    ├── README.md
    ├── pao
    │   ├── nah.py
    │   ├── __init__.py
    │   └── allow.py
    ├── setup.py
    └── tests.py

3 directories, 20 files
:~/gitexp/abc$ 

I searched but answers like this didn't help. I'm using git 2.11.0. To be sure that the procedure for merging two repositories can be done on my system, I've experimented creating two empty repositories, "repo1" and "repo2", then merged both into a "super" repo. You can see the commands and output below.

:~$ mkdir super
:~$ cd super
:~/super$ git clone ~/repo1/


Cloning into 'repo1'...
done.

:~/super$ git clone ~/repo2

Cloning into 'repo2'...
done.

:~/super$ ls
repo1  repo2

:~/super$ cd repo1/
:~/super/repo1$ git filter-branch --index-filter 'git ls-files -s | sed "s-\t-&repo1/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Rewrite 5c81fdf85fbd794efd4c52e339ed101c37400eb5 (1/1) (0 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/master' was rewritten

:~/super/repo1$ cd ../repo2
:~/super/repo2$ git filter-branch --index-filter 'git ls-files -s | sed "s-\t-&repo2/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Rewrite 80b1ece2a855d173717c02e093182e236c3506be (1/1) (0 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/master' was rewritten

:~/super/repo2$ cd ..

:~/super$ git init
Initialized empty Git repository in /home/nagev/super/.git/
:~/super$ git pull repo1
remote: Counting objects: 4, done.
remote: Total 4 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From repo1
 * branch            HEAD       -> FETCH_HEAD
:~/super$ rm -rf repo1/repo1
:~/super$ rm -rf repo1/.git/

:~/super$ git pull repo2 --allow-unrelated-histories
warning: no common commits
remote: Counting objects: 4, done.
remote: Total 4 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From repo2
 * branch            HEAD       -> FETCH_HEAD
Merge made by the 'recursive' strategy.
 repo2/file2 | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 repo2/file2

:~/super$ rm -rf repo2/repo2
:~/super$ rm -rf repo2/.git
:~/super$ 
:~/super$ tree
.
├── repo1
│   └── file1
└── repo2
    └── file2

2 directories, 2 files

:~/super$ git log
commit 05eaf17bf5d8fe8c3e7128ddbd00a220cedd0867
Merge: 2f5d0aa 301e92d
Author: Nagev <nagev>
Date:   Tue Feb 21 15:14:34 2017 +0000

    Merge repo2

commit 301e92d8eb97c1a8b6bdaded89a279a419042bb0    
Author: Nagev <nagev>
Date:   Tue Feb 21 15:10:47 2017 +0000

    Add file Two

commit 2f5d0aa4d5104c7b88564bf3d13cc75cfbf9a119
Author: Nagev <nagev>
Date:   Tue Feb 21 15:09:57 2017 +0000

    Add File One

As you can see that works fine. Any idea what could possibly be messing up repo "abc"?

Answer 1

I had this error message on checking out a different branch. It was quite frustrating.

Do git clean -xdfn abc/abc

If you are OK with deleting all the listed files, then do git clean -xdf abc/abc

Now see if that solves your problem.

Apparently, git ignores these files (as per gitignore), but refuses to overwrite/delete them.

Answer 2

First, I recommend avoiding git pull. It is a bad tool, a sort of Swiss army knife with the corkscrew and toothpick permanently stuck out at weird angles that will cut you if you are not careful. :-) It essentially just runs git fetch for you, followed by running a second command—usually git merge—for you. You should just run these yourself. However, it also has a special case for pulling into an empty repository. This is where the corkscrew is poking you. A real merge would fail too, but if so, you could read up on when and how and why git merge fails, and find these things.

In any case, the part that is failing is the second step. As Git says, it fails because you have, in the current working tree, files named abc/abc/something, and these files are not in the current index. That's what makes a file untracked: a file is untracked if and only if it is is not in the current index.

Your current work-tree is the directory gitexp, including all its contents. Its contents include two sub-directories abc and good. The fact that these sub-directories contain Git repositories does not matter here: they are still sub-directories with files in them.

Your current index is empty, because you are in a new, empty repository. Your work-tree should be empty, but it isn't. This is the main root of your problem. You can fix that by not working in a directory above the two sub-directories. For instance, if you start with:

:~$ mkdir gitexp
:~$ cd gitexp/
:~/gitexp$ git clone https://github.com/nagev/good

then, by the time you reach the point where you wish to combine the two filtered repositories, instead of:

:~/gitexp/abc$ cd ..
:~/gitexp$ git init

you might use:

cd .. && mkdir combined && cd combined && git init

This way you will start with an empty work-tree, as you should.

How to avoid git pull here

(You can ignore the rest of this if you want, but it will probably tell you some things you did not realize.)

While this should make the problem go away—you can now use git pull successfully twice, although obviously you will need slightly different path names to the two existing repositories—it's still wisest to avoid git pull. Remember, git pull generally means git fetch followed by git merge (of whatever you just fetched). But you are starting with an empty repository—there is literally nothing to merge! What, precisely, do you intend to accomplish by merging "nothing" with "something"?

I'm not sure what you intend, but the special case that git pull implements is to, in effect, act as if you have cloned only the "something", and made it your current (in this case, master) branch. You might as well just clone the first repository: the effect will be the same, except that you will get all the branches (as remote-tracking branches), instead of just one. To get just one branch, use --single-branch. (In this case, since there is only one branch, there won't be much difference in the end. While --single-branch affects the fetch refspec for the remote, we're about to delete the remote anyway.) Then you can delete the remote itself.

In other words, to make a clone into which you will add the second repository's commits:

cd .. && mkdir combined && cd combined
git clone --single-branch master ../abc .
git remote remove origin

or—and this might be superior in a few ways (fewer commands), but it's not as obvious where the mkdir happens for the new copy of abc; it's during git clone:

cd .. && git clone --single-branch master abc combined
cd combined
git remote remove origin

Now you can git fetch one branch from the other repository, and then merge:

git fetch ../good master:good
git merge good
git branch -d good

That is, you fetch the master branch from the repository in ../good, calling it a new branch good; then merge this branch, and then delete it. (That's essentially what git pull would do.) You may need to add --allow-unrelated-histories to the git merge command, depending on how new your Git is, because the good and master branches are not related.

Based on your filter-branch output, your repository named good has 60 commits on its master, and your repository named abc has 38 commits on its master. I won't draw all of them, but in graph form, these commits, now fetched into the new combined repository, look a bit like this:

o--o--... 38 little round o nodes total ...--A            <-- master

o--o--... 60 little round o nodes total ...--o--o--o--B   <-- good

To merge these, Git will have to compare what changed between the common base commit and the two branch tips. But there is no common base commit. What git merge does, which now requires the --allow-unrelated-histories flag, is to compare each tip commit—the ones I labeled A and B here—to an empty tree. That is, each commit is considered to have added its snapshot to a commit that had no files at all. This "no files at all" commit is the pretend-common-base.

(It's quite adequate and appropriate as this common base, too. After all, the very first commit on master consists of adding all-new files to nothing, and so does the very first commit on good. An empty tree is a valid common base! It's what Git used to use automatically for this merge case. But it's surprising; hence the new flag.)

As long as all the files have different names—and after your filter-branch commands, they do—the merge will succeed, and make a new merge commit:

o--o--...--o--A-------o   <-- master
                     /
o--o--...--o--o--o--B     <-- good