How to remove files that are listed in the .gitignore but still on the repository?

Question

I have some files in my repository that should be ignored, i added them to the .gitignore but, of course, they are not removed from my repository.

So my question is, is there a magic command or script using filter-branch that can rewrite my history and remove all these files easily? Or simply a command that will create a commit that will remove them ?

Answer 1

A simple solution to remove files already committed but now in .gitignore, is in the following commands:

git rm --cached `git ls-files -ic --exclude-from=.gitignore`
git commit -m "Remove ignored files"

This will remove the files from the repository (keeping them in your working directory) and create a commit for the changes.

Answer 2

You can remove them from the repository manually:

git rm --cached file1 file2 dir/file3

Or, if you have a lot of files:

git rm --cached `git ls-files -i -c --exclude-from=.gitignore`

But this doesn't seem to work in Git Bash on Windows. It produces an error message. The following works better (-d '\n' to split input line by line):

git ls-files -i -c --exclude-from=.gitignore | xargs -d '\n' git rm --cached  

If you are using PowerShell on Windows this works too (handles spaces in path and filenames):

git ls-files -i -c --exclude-from=.gitignore | %{git rm --cached $_}

Regarding rewriting the whole history without these files, I highly doubt there's an automatic way to do it.
And we all know that rewriting the history is bad, don't we? :)

Answer 3

I've come up with a solution that leverages git rm's --pathspec-from-file option. It's a few more steps, but it avoids some platform-specific pitfalls.

1. Write out the list of files to be removed to a temporary file (you can call this file whatever you like - I've used ignoreme.tmp here)

   This is all the files that should be ignored per .gitignore but are currently tracked in git

git ls-files -icX .gitignore > ignoreme.tmp

2. Remove the files using rm (you can use the -n option to do a dry run to see what files would be removed)

git rm -r --cached --pathspec-from-file=ignoreme.tmp

3. Remove the temporary file when you're done

del ignoreme.tmp

...or, for Linux / Unix

rm ignoreme.tmp

4. Commit your changes, as usual

git commit -m "Remove files that should be ignored"

tested on git version 2.35.1.windows.1, but should apply just as well to newer versions / other platforms

Answer 4

An easier way that works regardless of the OS is to do

git rm -r --cached .
git add .
git commit -m "Drop files from .gitignore"

You basically remove and re-add all files, but git add will ignore the ones in .gitignore.

Using the --cached option will keep files in your filesystem, so you won't be removing files from your disk.

Be sure to do this from a clean working directory, i.e. commit your other diffs before doing this.

Note: Some pointed out in the comments that you will lose the history of all your files. I tested this with git 2.27.0 on MacOS and it is not the case. If you want to check what is happening, check your git diff HEAD~1 before you push your commit.

This won't remove the .gitignor'ed files from the git history of your repo. If you want to do that, check Ignore files committed to git and also remove them from history.

Answer 5

For GitHub, the easiest way is to open the online VS Code editor by pressing . when viewing the repo. Then you can just right-click and delete the files/folders in the left hand panel and commit the change.

Answer 6

Simply go to gitignore file and remove whatever added newly and then commit the changes, it remove contents from there

Answer 7

If you really want to prune your history of .gitignored files, first save .gitignore outside the repo, e.g. as /tmp/.gitignore, then run

git filter-branch --force --index-filter \
    "git ls-files -i -X /tmp/.gitignore -c | xargs -r git rm --cached --ignore-unmatch -rf" \
    --prune-empty --tag-name-filter cat -- --all

Notes:

• https://git-scm.com/docs/git-filter-branch#_warning is not just there for fun
• git filter-branch --index-filter runs in the .git directory I think, i.e. if you want to use a relative path you have to prepend one more ../ first. And apparently you cannot use ../.gitignore, the actual .gitignore file, that yields a "fatal: cannot use ../.gitignore as an exclude file" for some reason (maybe during a git filter-branch --index-filter the working directory is (considered) empty?)
• I was hoping to use something like git ls-files -iX <(git show $(git hash-object -w .gitignore)) instead to avoid copying .gitignore somewhere else, but that alone already returns an empty string (whereas cat <(git show $(git hash-object -w .gitignore)) indeed prints .gitignore's contents as expected), so I cannot use <(git show $GITIGNORE_HASH) in git filter-branch...
• If you actually only want to .gitignore-clean a specific branch, replace --all in the last line with its name. The --tag-name-filter cat might not work properly then, i.e. you'll probably not be able to directly transfer a single branch's tags properly

Answer 8

I did a very straightforward solution by manipulating the output of the .gitignore statement with sed:

cat .gitignore | sed '/^#.*/ d' | sed '/^\s*$/ d' | sed 's/^/git rm -r /' | bash

Explanation:

1. print the .gitignore file
2. remove all comments from the print
3. delete all empty lines
4. add 'git rm -r ' to the start of the line
5. execute every line.

Answer 9

As the files in .gitignore are not being tracked, you can use the git clean command to recursively remove files that are not under version control.

Use git clean -xdn to perform a dry run and see what will be removed.
Then use git clean -xdf to execute it.

Basically, git clean -h or man git-clean(in unix) will give you help.

Be aware that this command will also remove new files that are not in the staging area.

Answer 10

git rm --cached -r . to remove all cached recursively

git add . to add all files not included in .gitignore

you will have to commit some deleted files that are not really deleted on File System

using a single command it will be git rm --cached -r . && git add .

Answer 11

This solution adds carriage returns (I'm a WSL user, so this is important), and parenthesis escaping (which is important to LaTeX users sometimes, e.g. *.synctex(busy)).

---

Inspired by Scott's solution:

cat .gitignore | sed "s/\r//" | sed -r "/^(#.*|\s*)$/d" | sed -r "s/([()])/\\\\\1/g" | sed "s/^/git rm -r /" | bash

1. Remove: carriage returns (s/\r//).
2. Remove lines containing: comments (/^#.*$/), empty line groups (/^\s*$/, matches whitespace or empty line). Notice the pipe | character, this is standard regex, and requires -r (although I believe -E also works).
3. Replace: parenthesis /([()])/ with its escaped version \\\1, \1 matches the group, in this case it means [()], or ( or ), whatever was matched. Notice the g flag, this is to match (and replace) all parenthesis. Could be rewritten as "s/(\(|\))/\\\\\1/g" if you're into that.
4. Prepend git rm -r

Replacement looks like s/$old/$new/$flags. Removal looks like /$old/d. Prepending is replacing /^/. And you could do appending by replacing /$/. And of course, some characters are escaped, since you can't make raw strings in bash as far as I know. Finally, this line can be condensed, but I chose to leave it expanded for the sake of readability.

---

I saw someone questioning (in Scott's solution) that sed is straight forward. I like to think of this method as the most basic and most monkey-way to do it, this is good, because if you ever need a variation of this, you can make it on the spot. And if anything, it's a good excuse to practice regular expressions.

Answer 12

"git clean"^(man) and git ls-files -i^(man) had confusion around working on or showing ignored paths inside an ignored directory, which has been corrected with Git 2.32 (Q2 2021).

That means the 2021 version of the accepted answer would be:

git ls-files -i -c --exclude-from=.gitignore | xargs git rm --cached  
                ^^

See commit b548f0f, commit dd55fc0, commit aa6e1b2, commit a97c7a8, commit 2e4e43a, commit b338e9f, commit 7fe1ffd, commit 7f9dd87 (12 May 2021) by Elijah Newren (newren).
See commit 4e689d8 (12 May 2021) by Derrick Stolee (derrickstolee).
^{(Merged by Junio C Hamano -- gitster -- in commit 33be431, 20 May 2021)}

ls-files: error out on -i unless -o or -c are specified

^{Signed-off-by: Elijah Newren}

ls-files --ignored^(man) can be used together with either --others or --cached.

After being perplexed for a bit and digging in to the code, I assumed that ls-files -i was just broken and not printing anything and I had a nice patch ready to submit when I finally realized that -i can be used with --cached to find tracked ignores.

While that was a mistake on my part, and a careful reading of the documentation could have made this more clear, I suspect this is an error others are likely to make as well.
In fact, of two uses in our testsuite, I believe one of the two did make this error.
In t1306.13, there are NO tracked files, and all the excludes built up and used in that test and in previous tests thus have to be about untracked files.
However, since they were looking for an empty result, the mistake went unnoticed as their erroneous command also just happened to give an empty answer.

-i will most the time be used with -o, which would suggest we could just make -i imply -o in the absence of either a -o or -c, but that would be a backward incompatible break.
Instead, let's just flag -i without either a -o or -c as an error, and update the two relevant testcases to specify their intent.

That means without -c, you would get (starting with Git 2.32, Q2 2021):

fatal: ls-files -i must be used with either -o or -c

Note: this is still a work in progress, since it was reverted in Git 2.32-rc2 but fixed with commit 2c9f1bf, commit 1df046b (27 May 2021) by Junio C Hamano (gitster).
See commit 906fc55 (27 May 2021) by Elijah Newren (newren).
See commit eef8148 (27 May 2021) by Derrick Stolee (derrickstolee).
^{(Merged by Junio C Hamano -- gitster -- in commit 329d63e, 28 May 2021)}

dir: introduce readdir_skip_dot_and_dotdot() helper

^{Signed-off-by: Elijah Newren}

Answer 13

In linux you can use this command:

For example I want to delete *.py~ so my command should be ==>

find . -name "*.py~" -exec rm -f {} \;

Answer 14

The git will ignore the files matched .gitignore pattern after you add it to .gitignore.

But the files already existed in repository will be still in.

use git rm files_ignored; git commit -m 'rm no use files' to delete ignored files.