How to remove binary files from all of the bitbucket git history, when the repo is already very close to 2GB?

Question

Bitbucket git repo has a size limit of 2GB, and now I have one repo ( let's call it bigsize repo) that is already dangerously close to that limit, due to a lot of binary files ( files with extensions of dll and msm). It's so close to the limit that I'm scared anymore commit operation involving binary files will tip the size to over 2GB, and hence the commit will fail and unable to proceed.

Now, how to best go by to reduce the bigsize repo size?

I'm thinking about using the LFS feature, but then, there is a 1GB limit on the LFS space, which I afraid will not be sufficient for the usage of this repo ( as the majority of the repo size is coming from the binary files that I want to store in LFS).

So I'm thinking about just removing all the binary files ( I don't mind losing them from source control, as I have them on my local drive) from the repo and the history, how to best do this, considering the current size of my bigsize repo?

The attack plan that I have:

1. Make sure that for all the branches on the bigsize repo ( yes, I've more than one branch on this gigantic repo), I've remove all of the binary files ( by submitting a commit that specifies *.dll in gitignore and use git rm -rf -cached command) . This is needed because "By default the BFG doesn't modify the contents of your latest commit on your master (or 'HEAD') branch, even though it will clean all the commits before it."
2. Then use BFG delete-files command to "rewrite the history" so that the repo size will be reduced.

Does the approach work for a repo that is very closed to 2GB? I afraid that at step 1, when I use git rm, it will add to the history and push the repo size to over 2GB, and hence fail.

Important details:

1. I'm the sole author of the repository
2. Now I am multiple active branches. Throughout the history there are multiple branches that are merged into the main branch
3. I don't use the repo for discussions or code reviews, and not even tags. I just use it as a single branch, with occasional branching and merging

Answer 1

Here's how I solve the problem using BFG repo cleaner. I write it down so that it will be helpful to those in the future who might face the same situation as I do.

Before one even gets started, make sure that all your branches are "clean", ie: all changes that you desired are pushed into their respective branch. Let's name our gigantic repo as big repo, and assume that it is located at D:\MyCompany\big.

Also you might want to try this on a toy repo before actually working on the production repo.

1. Clone your big repo into a brand new folder (git clone --mirror git@bitbucket.org:YourUserName/big.git), make sure that this new folder is outside of your original big repo folder. You need to be very sure that this is a completely separate thing from D:\MyCompany\big ( and hence you can experiment on it without fear). Assuming that this folder is D:\big_work, you will be able to find D:\big_work\big.git folder after doing the cloning.
2. Backup big.git folder to another place, let's put it in D:\backup.
3. determine that your binary files are really out of the LFS limit provided by bitbucket ( or your repo provider). In my case here, I need to ascertain that my binary file size is less than 1GB. To do this, convert the git of big.git to LFS via BFG. At command line:

3.1. cd D:\big_work

3.2. java -jar <path to>bfg-x.x.x.jar --convert-to-git-lfs "*.{dll, msm}" --no-blob-protection big.git ( make sure that you do this while you are inside the D:\big_work folder)

4. Now check the size of D:\big_work\big.git\lfs folder, is it more than 1GB? If no, you can proceed to convert the binary files to lfs by continuing from step 4 here. No need to read the rest.
5. If the lfs folder is more than 1GB, then you have to delete binary files from your repo and rewrite the history.
6. Delete the big.git folder from D:big_work, and restore it from D:\backup ( the backup that you did in step 2).
7. Here comes the most important step: the actual deletion. A lot of guides talk about why it's important to first make sure that the commits are clean (meaning, there are no more binary files in current state of repo) before carrying out this step. But I do find advice such as this unnecessary and distracting. Since we already know very well what we want-- namely, remove all of the binary files from our repo, present and past, from all branches, merged or unmerged, and the repo should look like as if it was never contaminated by binary files at any one point of time-- we should just disregard this warning. We bring on our nuclear weapon no-blob-protection for this purpose.

7.1. cd D:\big_work

7.2. java -jar <path to>bfg-x.x.x.jar --delete-files "*.{dll,msm}" --no-blob-protection big.git

8. cd D:\big_work\big.git
9. git reflog expire --expire=now --all && git gc --prune=now --aggressive
10. git push
11. Now go to your bitbucket big repo, you should find that all your dlls and msms are removed from all branches, and all history. This should give you the confidence that you already did the right thing: the binary files are being removed and your repo saved!
12. Remember that your original big repo is located at D:\MyCompany\big and you have your sourcetree bookmark pointed to this folder, so you will need to first remove the sourcetree bookmark, and rename the D:\MyCompany\big to D:\MyCompany\big_old. Don't delete big_old folder yet because even though you don't want the binary files to locate inside the repo, but you still want them to locate on your hard disk, and big_old folder provides a natural backup. You can delete big_old folder some point in the future, after you are sure that it is really of no value.
13. You need to clone again the clean repo to your hard disk. Let's clone it back to D:\MyCompany\big folder to avoid any breaking changes.
14. Repoint the sourcetree big repo bookmark to D:\MyCompany\big folder.
15. Gradually transfer the binary files you need from big_old to big folder on need basis, from time to time.
16. You might need to contact bitbucket support for them to run agarbage collection operation to reduce the size.

Main references:

1. BFG for a Noob
2. Use BFG to migrate a repo to Git LFS
3. BFG repo cleaner

Answer 2

git filter-branch or BFG are obsolete

With Git 2.22 or more, use git filter-repo:

git filter-repo --path your/big/file --invert-paths

Or:

git filter-repo --strip-blobs-bigger-than 10M