Error with Previously Committed Files Which Matches New Git LFS Filter

Question

How to clean repo, if staged files marked as modified?

After

git reset --hard

I get

Encountered 7 file(s) that should have been pointers, but weren't:

Running git clean -fdx (which deletes everything, including untracked directories and anything in .gitignore) doesn't help, either.

Answer 1

None of the suggestions above worked in my case. What I did was something much simpler. Cleaning, uninstalling git lfs, once uninstalled I was able to pull the remote branch and then I installed the lfs again, pulled lfs and all was back to normal.

git clean -fdx ./
git lfs uninstall
git pull
git lfs install
git lfs pull

I hope it helps.

Answer 2

I removed every file manually from cache and checkout to head, after that did reset --hard

git lfs uninstall
git rm -r --cached FILE_THAT_SHOULD_HAVE_BEEN_A_POINTER_BUT_WASNT  //run this individually for every file showing in git status that is not pointed
git checkout HEAD
git reset --hard
git lfs install
git lfs pull

Answer 3

In addition to the karyon's answer, after performing migrate to fix the local state:

git lfs migrate import --no-rewrite "broken file.jpg" "another broken file.png" ...

You have an extra commit which you probably want to get rid of.

Considering .gitattributes is already fixed on you main / master branch, one can just rebase the local branch onto the one containing the fix, usually:

git rebase upstream/master

And this extra commit will be dropped automatically, since it is meaningless at this point.

Answer 4

A lot of these solutions appear to really be overkill.

Here's what worked for me:

git lfs uninstall
# go into .gitattributes and comment out the whole file
git rm -r --cached FILE_THAT_SHOULD_HAVE_BEEN_A_POINTER_BUT_WASNT
git checkout HEAD -- FILE_THAT_SHOULD_HAVE_BEEN_A_POINTER_BUT_WASNT

I have 2 scripts in my .zprofile to help me with this:

#aka fix "pointers that weren't"
untrack() {
  git rm -r --cached $1
}

resetfile() {
  git checkout HEAD -- $1
}

so you could just call

untrack "FILE_THAT_SHOULD_HAVE_BEEN_A_POINTER"
resetfile "FILE_THAT_SHOULD_HAVE_BEEN_A_POINTER"

Don't forget to reinstall git-lfs!

Answer 5

Going off Pellet's answer, I found that sleeps even up to 10s sometimes weren't working. Instead, a single call to git status before git reset . always worked without needing any sleeps, probably thanks to some kind of refreshing that happens.

The git lfs uninstall and git lfs install commands from the accepted answer were also necessary for me. Combining it all:

resetlfs() {
  pushd "$(git rev-parse --show-toplevel)" > /dev/null

  git lfs uninstall

  git rm --cached -r .
  git reset --hard
  git rm .gitattributes

  git lfs install
  git status

  git reset .
  git checkout .

  popd > /dev/null
}

Note that this approach only hides the .gitattributes violations (useful if you're just trying to switch branches!), whereas karyon's answer resolves them permanently.

Answer 6

Since git lfs 2.5.0, there is a new command available that makes this easier (docs):

git lfs migrate import --no-rewrite "broken file.jpg" "another broken file.png" ...

This "migrates" files to git lfs which should be in lfs as per .gitattributes, but aren't at the moment (which is the reason for your error message).

--no-rewrite prevents git from applying this to older commits, it creates a single new commit instead.

Use -m "commitmessage" to set a commitmessage for that commit.

Answer 7

What helped me, without touching the whole repo, was the git restore command which was added in git 2.23

git restore --source=HEAD --staged --worktree -- affected_files

Execute this command a few times until all of the warnings are gone.

Answer 8

Here with this embellishment that doesn't require reinstalling the lfs hooks or obscuring any .gitattributes files:

git -c filter.lfs.smudge= -c filter.lfs.clean= reset --hard

Answer 9

Loads of answer on this with multiple steps to fix.

If you are just in a broken state, like stuck on a branch because you can't reset/discard the changes to the problem file(s): deleting the .gitattributes file maybe enough to allow you to make your next git move. Once you have made your git move you may then have to restoring the .gitattributes file but at least you are unstuck.

I wished I had known this before trying all the above. It is a low risk option to try at least.

Answer 10

Analysis

That's because the files are not tracked by LFS, but they matches some .gitattributes files' description.

For example,

server/.gitattributes:

conf/** filter=lfs diff=lfs merge=lfs -text

• server/conf/client.conf is too large and is tracked by LFS
• server/conf/client.gflags is tracked in git instead of LFS

However, client.gflags matches the server/.gitattributes description, and git will pull it from LFS, but it doesn't have the LFS info, and the error will be thrown up.

Solution

Find the .gitattributes file whose description hit the Encoutered file, delete the wrong description or optimize some wildcard character match.

Optimize the example above, server/.gitattributes:

conf/client.conf filter=lfs diff=lfs merge=lfs -text

Answer 11

Same as @John Kugelman above, but I put it in a alias because I had to do it a bunch of times.

    
git rm --cached -r . > /dev/null && git reset --hard > /dev/null && git rm .gitattributes > /dev/null && git reset . && git checkout . > /dev/null

Answer 12

Ensure you have git lfs version 2.5 or higher installed (download here).

Check you are using the git lfs version you downloaded (2.7.2 for me):

>git lfs version
git-lfs/2.7.2

Run:

git lfs migrate import --fixup --everything

Pull your branch and fix any merge conflicts.

Found in this github comment.

Answer 13

The problem comes from the mismatch beetween the filetypes marked as to be tracked by git LFS in the .gitattributes and some matching files already under conventional non-LFS version control.

So the simplest workaround here is to just remove the .gitattributes file for a moment:

git rm .gitattributes
git reset .
git checkout .

Afterwards you can checkout any other branch.

One more advice: When adding a new filetype to git LFS, prefer not to do this by hand by modifying the .gitattributes but e.g. by running:

git lfs track PATTERN

where PATTERN is the pattern for matching files, e.g *.so

This way all already non-LFS versioned files matching the new tracking-pattern will be marked dirty and can be simlpy added, i.e. converted to git LFS (file pointers).

Answer 14

In my case it was one file under lfs rules (I'm assuming it was checked in without lfs installed or something).

So I found it's extension in the .gitattributes file and commented this line out like

#*.7z filter=lfs diff=lfs merge=lfs -text

saved this .gitattributes then git status showed no problems.

After that I've uncommented this line (removed #), saved .gitattributes and git status still shows no problems.

Answer 15

This just shows one more time what a pile of dog**** GIT-LFS is.

You can get into this situation if:

1. The file is contained in common-base-branch and not in LFS
2. In a branch lfs-branch based on common-base-branch, the file was moved to LFS
3. In another branch non-lfs-branch also based on common-base-branch, the file was modified.

Or alternatively:

1. The file is NOT contained in common-base-branch
2. In a branch lfs-branch based on common-base-branch, the file was added to LFS
3. In another branch non-lfs-branch also based on common-base-branch, the file was added (but not to LFS.

In both cases, when you try to merge non-lfs-branch into lfs-branch, you get this kind of error.

You may ask why this should even happen in the first place, but the answer is that a lot of software is developed by more than once person (which is why you have version control systems like GIT in the first place), and people don't always talk to each other, or LFS is introduced later in the history of a project in a feature branch, while "normal" development still goes on in other branches.

This is a legitimate merge conflict situation, not a bug or corrupted working directory or anything (as some of the other answers suggest). GIT-LFS just handles it poorly.

What you want to do now is to make sure the right version of the conflicted files goes into GIT-LFS, so you may want to choose an answer to this question which does just that... (TODO: Insert link to at least one answer that works)

Answer 16

Neither of these solutions worked for me, but I pieced together a few sources to finally get all this fixed.

1. Push any changes you don't want to lose

   If you can... If not, or if you don't care about your changes, press on.

2. Stop Everything

   SourceTree, any servers, file explorers and browsers. Sometimes this stuff won't work if it's being used somewhere else. When in doubt, stop it - with this it's better to overkill.

   Also, go into Task Manager, force quit any bash.exe processes. Git Bash tends to hold files open after you close the window.

3. Open a Command Window (or Terminal)

   cd to your local repo.

4. Uninstall lfs

   > git lfs uninstall

   Then it'll say something like:

   Hooks for this repository have been removed.
   Global Git LFS configuration has been removed.
   

5. Reset

   > git reset --hard

   It'll go through a lot of output probably...

6. Reinstall lfs

   > git lfs install

   This may again say it found files that should have been pointers but weren't. That's OK, keep going!

7. Pull with lfs

   > git lfs pull

   Hopefully pulling with lfs will overwrite the files that got borked.

   A few of my sources said at this point their repo was working again, but not me personally. You can open SourceTree or whatever to check if you want, but you may have to start from the top if it didn't work.

8. Migrate

   The core issue here is that lfs, instead of downloading large files like audio, video, images - anything larger than 1Mb - it just points to them on a server. This is useful if you have a bunch of large files, you're not pulling down all that stuff. So your local repo is smaller and nimbler. However, through circumstances I'm not sure about, it seems possible to corrupt the pointers. I'm sure this is an issue that the lfs people are aware of and are working on, but for now we have to work it out ourselves.

   What we've done so far is

   • uninstall lfs
   • delete everything
   • reinstall lfs
   • pull everything

   So now we have all these things in our folder that are either files or pointers to files, and lfs needs to figure out if any files should be pointers and vise versa. And hopefully by performing the steps above we deleted the corrupted pointers. So we're going to perform migrate to kick off the procedure that goes through the files on the repo, and if they're greater than 1Mb, lfs is going to replace them with a pointer.

   > git lfs migrate

9. More Errors

   Here's a point at which others have stopped and said they were working again, but not me. I got an error:

   Error in git rev-list... exit status 128 fatal: bad revision '...v1.0.0'

   @guneyozsan over at a github help page, posted this final piece to the puzzle, even though it didn't fix his issue.

   > git lfs migrate info --include-ref=v1.0.0

   Notice the version matches the version that errored - v1.0.0. You will need to replace v1.0.0 with whatever version you got in your error.

   I haven't found a source on why this error occurs but my guess is that the lfs version number generated by migrate on your local repo doesn't match the source version. For me, all this started when SourceTree crashed during a push and I forced a machine reboot, and when that happens, lfs doesn't know how to deal with it, so it just gets stuck in this loop where it's trying to update, but it can't read the corrupted data. Hence the lengthy troubleshooting.

10. Stash and Pull

When you open SourceTree, you'll probably see that it wants to add all your files back. Don't do that. Stash, then pull.

And boom, the horror is hopefully over. If not, this git hub page or this one may help you more, but this is what worked for me.

Answer 17

run

git add --renormalize .

and commit those changes. It is safe to do even when another user is doing the same on another branch, since the LFS pointer is derived from the hash of the file. It might also catch some files with wrong line endings.

Answer 18

none of the above commands worked for me, I found an answer here

git status -s | cut -c 4- | xargs git update-index --assume-unchanged
rm .git/index && git reset

Answer 19

The accepted answer worked for me but it would only work when I manually typed the commands, I put sleeps between each command and now it works as a bash script:

git rm --cached -r .
sleep 1
git reset --hard
sleep 1
git rm .gitattributes
sleep 1
git reset .
sleep 1
git checkout .

Answer 20

Here's the problem I ran into:

Say you created a branch, and you somehow committed files as non-LFS. So then you tried to correct it by later committing the LFS version of the files on the same branch. However, now you can't rebase or squash because you'll keep running into this "files that should have been pointers but weren't" error in the middle of the rebase.

Solve using git reset --soft: https://stackoverflow.com/a/5201642/2516916

Answer 21

This can happen when you do a checkout that contains files which should have been been tracked by LFS as specified in .gitattributes but somehow they've been committed directly instead. Most likely you have another program managing your repository such as a git GUI or IDE.

This can be frustrating because these files appear out of nowhere and prevent you from making checkouts. As soon as you stash your changes they return! If you get stuck in this situation, a quick fix is to commit these changes on a temporary branch so that you can checkout again.

To genuinely fix this problem, make sure you've committed the files as LFS pointers. This should be as simple as using git add. Check your work using git lfs status before committing. git lfs ls-files will show what files LFS is managing.

git lfs status is misleading since it reads Git LFS objects to be committed when it really lists all changes. Files that you expect to be tracked by LFS should read something like (LFS: c9e4f4a) or (Git: c9e4f4a -> LFS: c9e4f4a) and not (Git: c9e4f4a).

By way of example, I found this to be a problem when adding image assets through Xcode 9.2 where I added "CalendarChecked.png" which it automatically added:

$ git status
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    new file:   Example/Assets.xcassets/CalendarChecked.imageset/CalendarChecked.png

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   Example/Assets.xcassets/CalendarChecked.imageset/CalendarChecked.png

$ git lfs status

Git LFS objects to be committed:

    Example/Assets.xcassets/CalendarChecked.imageset/CalendarChecked.png (Git: c9e4f4a)

Git LFS objects not staged for commit:

    Example/Assets.xcassets/CalendarChecked.imageset/CalendarChecked.png (File: c9e4f4a)

$ git add Example/Assets.xcassets/CalendarChecked.imageset/CalendarChecked.png`
$ git lfs status

Git LFS objects to be committed:

    Empty/Empty/Assets.xcassets/CalendarChecked.imageset/CalendarChecked.png (LFS: c9e4f4a)

Git LFS objects not staged for commit:

$

Answer 22

If you just want to get away from that bad commit, you can go back to master by

git reset --soft origin/master
git reset --hard

Then you are free from the nasty 7 non-LFS files:-)

Answer 23

Like Travis Heeter mentioned in his answer, Try the following command sequence:

git lfs uninstall
git reset --hard
git lfs install
git lfs pull

In case if this is not working (because this was not working for me), the following hack may work:

git rm --cached -r .
git reset --hard
git rm .gitattributes
git reset .
git checkout .

This worked for me!

Answer 24

One possible cause for this error is due to git LFS-related changes to .gitattributes that impacts already added files in a repo.

(I'm not sure of the exact steps to reproduce, but the issue seemed to occur when I touched a file that was newly affected by the .gitattributes that was previously committed as a non-LFS file that should now be an LFS file. The issue seemed aggravated by switching branches, or at least made it impossible to switch branches until the issue was resolved.)

In this case, I used the steps below to prevent this error from occurring repeatedly.

---

1. Fix the issue for the branch you are on by following one of the other answers here (e.g. to clear the cache and reset. I found BheeMa's answer to be effective.)
2. Go to your main branch, and ensure there is nothing to commit with git status
3. Force git to recheck and "reapply" git attributes changes

From Ratata Tata's answer in How to make change to .gitattributes take effect)

 git rm --cached -r .
 git add -A

Warning: make sure in step 2 that there was nothing to commit, as the steps above will add any files that were not previously versioned

4. Verify the results with git status (it should only have modified relevant files to become LFS pointers, i.e. files that can potentially cause the "encountered files that should have been pointers" error) and commit changes
5. (Optionally, merge/rebase this fix to all other branches if possible. Otherwise, this error could pop up again when switching to those branches. Note that it may be necessary to repeat the initial fix for each branch as per step 1 to be safe, though it could be ok just to commit the affected files.)

Answer 25

The following process will add a commit which replaces all binary files that should be lfs pointers with lfs pointers.

1. Clean working copy completely. Together with the force add below this prevents any files getting added or removed due to .gitignore patterns.

   git clean -dfx
   git reset --hard
   git checkout -- .
   

2. Add remove for everything to staging area. Working copy will not be touched.

   git rm --cached -r .
   

3. Readd all files from working copy again. This will basically undo the previous command but will reevaluate lfs filters. Use -f to ignore .gitignore. All files present were previously checked in and should get added again.

   git add -f .
   

4. You staging area now should only contain the binary files that previously raised the 'should have been pointers' error.

   git commit -m "moved files to lfs"
   

Answer 26

When it is obviously an error that shows up out of nowhere, this is what we do in our team:

1. Disable lfs for that specific type file (modifying .gitattributes or via SourceTree menus)

2. The change will dissapear and you will see a change on .gitattributes instead

3. Remove the problem:

   3.1 One solution is to execute git reset --hard. Another way, discard changes. Sometimes the file will not come up again.

   3.2.1 If the previous solution doesn't work, repeat 1 and 2. Then make sure that this branch you are in (A) has already commited and pushed everything except those annoying files. Then commit your change, but not push.

   3.2.2: Go to another branch (B)

   3.2.3: Remove that local branch (A) where you performed the commit to .gitattributes, forcing the removal even when it says there's a commit that hasn't been pushed. It will forget that commit (it can afterwards be removed via GC or whatever but it's not a big deal if the file that has the error is not huge)

   3.2.4: Checkout the branch A again. It will download the previous status of the repository without the annoying files and LFS settings set up the correct way.

That always works!

Answer 27

I had this exact error with some files stored with git-LFS and solved it the same way I've solved a linending induced borked index .

Clear the cache and do a hard reset:

git rm --cached -r .
git reset --hard

This was significantly faster than a fresh clone for me due to the huge git-LFS files in my repo.