git rebase: The following untracked working tree files would be overwritten by checkout (though files are tracked)

Question

When attempting an interactive rebase, git says:

error: The following untracked working tree files would be overwritten by checkout:
        # some files here
Please move or remove them before you switch branches.
Aborting
error: could not detach HEAD

The set of files in question seem to vary depending on which commits I mark for (e)dit. They are all tracked files that have been modified along the commits in the branch, then finally removed in a subseqent commit.

Why is git doing this? How do I fix it?

Thanks!

Answer 1

Suppose you have :

• a history of commits where file foo.txt was modified, then deleted ;
• at the moment, next to your tracked files, you also have an untracked verion of foo.txt on your disk.

In this situation : git rebase will complain, beacause it does not know what to do with the current (untracked) file foo.txt without erasing its content.

Answer 2

Git is complaining because the files are untracked.

I know you said:

[the] files are tracked

but they aren't: you also said:

They are ... finally removed in a subsequent commit.

When you are at the point where the failure occurs, the files have been removed from the index, but not from the work-tree. That's the definition of an untracked file: it is a file that is in the work-tree, but not in the index.

Specifically, this:

error: could not detach HEAD

tells you (or me, anyway) that the files are currently untracked in the tip commit you have checked out right now, as you enter the command:

git rebase -i <start-point>

but the commit identified by <start-point> here has them in it. Git will need to git checkout that commit, but in checking out that commit, it will overwrite the untracked files with the tracked files.

If that last bit gave you a record-scratch / WTF noise in your head 😀 don't worry: it really is confusing.

The index, in Git, is a crucial little beastie that is often not explained properly. To understand it, you have to realize that Git has, at all times, three—well, up to 3—copies of each file that you're working on/with. One copy is frozen for all time, in a commit: the commit you have checked out right now. A second copy¹ lives in the index. It's what makes the file tracked. The third copy is the only one you can see ... but Git doesn't even use it as it's your copy, in your work-tree.

The index, which Git also calls the staging area, holds a copy (see footnote 1 again) of each file that will go into the next commit you make. As such, it's originally copied out of the commit you git checkout, as you pick a commit to work on. Git then copies the index copy, which is in a special Git format, out your work-tree, which is where Git puts all the snapshot files for you to see and work with.

Once Git has copied these out, it's mostly done with them. You make changes to your work-tree copy, and then run git add to tell Git: copy the work-tree copy back into / over top of the index copy. (Again, see footnote 1.) The reasons for this are a bit complicated, but if you like, think of it this way:

• The copies in commits are read-only, frozen for all time. To save a lot of space, Git keeps them in a special Git-only compressed format, that only Git can use.
• This makes sense for archival but is useless for getting work done, so Git has to expand out the file to its normal form. That's the work-tree copy.
• To make things go fast and make things convenient for Git, Git keeps the frozen copy (but not frozen any more) in the index. You can replace the index copy wholesale, by git add-ing from the work-tree: Git will get it into the freeze-dried format, ready to be committed, in the index (but it's still replaceable there).

You can do anything you want with the work-tree copy, because it's a regular file. As long as the work-tree file exists, it's just there, as a file.

You can remove or replace the index copy any time. To replace it, use git add to copy the work-tree file back over whatever is in the index, under that name. To remove it entirely, use git rm file.

When you run git commit, Git packages up all the files that are in the index to make the new commit's snapshot. Whatever is in the index—in whatever form it has in the index—goes into the new commit. Git doesn't use your work-tree for this at all. That's why you have to re-git-add files all the time: Git isn't looking at the work-tree copy.²

If the work-tree file exists and the index copy is gone, the file is untracked. But if the file is in some earlier commit and you check out that commit, the copy from the earlier commit goes back into the index, and the file is tracked again. And, when the copy from the earlier commit goes into the index, Git will overwrite the copy in the work-tree too—unless that would trash it because it's not saved anywhere, in which case, Git says that it's in the way.

(If it's tracked, Git knows if it's safely saved in a commit. If so, Git can overwrite it.)

So, what to do about this should be obvious now: if the data are important, move the files somewhere else. Do your rebase. When you're done, the files will be gone because the last commit removes them. Now you can move the files back and they're untracked again. The commits that had the files earlier still have the files now; if your rebase copied those commits to new-and-improved commits, those new-and-improved commits still have the files.

Key concept: whether a file is tracked or untracked depends on whether there is a copy in the index. The index copy is mostly invisible,³ but it's vitally important! The index is a temporary area. It mostly contains the next commit you will make; Git has to fill it in from a commit when you git checkout that commit.

Hence, the tracked/untracked state of some file can suddenly change. When Git is filling the index from a commit, it will create a file in the index if needed and create it in the work-tree at that time; or, it will remove a file from the work-tree if needed—if it's not in the commit you're moving to—and at that time, it will remove the file from the work-tree, too.

---

¹Technically, the index contains a reference to a blob object, for each file to be committed. The blob object is frozen for all time and commits refer to it, just like the index did. When you git add a file, Git compresses and hashes the content, and if there's already a blob with that data, Git can re-use the existing blob. Otherwise Git makes a new one and that blob's hash goes into the index.

Sometimes this creates a blob that never gets used, because you write one version in, then realize there's an error in it and write another one in. That's OK: Git creates garbage objects all the time, on its own. Git has a garbage collector that it runs, whenever it seems reasonable, to clean out unused objects.

²If you use git commit -a, that creates a second, but temporary index, git add-s all the files to the temporary index, and commits from the temporary index. If that all works, the temporary index becomes the new index; if it fails, Git simply removes the temporary index and everything goes back to however it was before you ran git commit -a. The TL;DR is that Git is still committing from an index. It's just a second, extra index for a bit.

³The git status command compares the index vs the HEAD commit and tells you what's different. So if a file has "gone missing", you'll see a deletion as staged for commit. Then, separately, it compares the index vs the work-tree and tells you what's different. So, if a file is in the index but missing from the work-tree, you'll see a deletion as not staged for commit.

If the file exists in HEAD, but is gone from the index and then is back in the work-tree, you'll see a staged deletion and an untracked file, both with the same name.