CODEWITHSUNDEEP
gitgithubgit-lfsgithub-enterprisebfg-repo-cleaner
Alberto
Alberto

Reputation: 5141

How to import git repositories with large files?

Given that GitHub doesn't allow to push files larger than 100 MB, it is not possible to git clone and push a repository with large files into GitHub enterprise. The push fails with a:

remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: File large.zip is 145.00 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB

(N.B.: there is a repository importer, but this is only for github.com, and requires public access to your repository)

Fortunately, GitHub provides support for storage of files larger than 100MB since April 2015. So how to convert a current repository with such large files into a GitHub LFS compatible repository I can push to?

Upvotes: 2

Views: 3842

Answers (2)

Ido Ran
Ido Ran

Reputation: 11414

You can now use git lfs migrate built-in command to both assess which files is best to migrate and also do the actual history-rewriting.

See git-lfs migration tutorial for more details.

Upvotes: 1

Alberto
Alberto

Reputation: 5141

The easiest way I found was taking advantage of git filter-branch and the BFG Repo-Cleaner by rtyley (I used version 1.12.12):

  1. Prerequisite: you need to have git lfs installed

  2. Create a new repository on GitHub Enterprise. You'll import your external Git repository to this new repository.

  3. Clone the repository you want to migrate to a local folder:

$ git clone --mirror git@oldgithost:repo
$ cd repo.git
# mirror into a local directory
  1. Rewrite the history to lfs-track your large files1:
$ git filter-branch --tree-filter 'git lfs track "*.{zip,jar}"' -- --all
# writes the patterns to lsf-track into .gitattributes
  1. Use the BFG to extract the relevant files into Git LFS
$ java -jar ~/usr/bfg-repo-cleaner/bfg-1.12.12.jar --convert-to-git-lfs '*.zip'
$ java -jar ~/usr/bfg-repo-cleaner/bfg-1.12.12.jar --convert-to-git-lfs '*.jar'
# Convert large files (I couldn't find a one-liner for multiple patterns)
  1. Push to your GitHub enterprise remote:
$ git push --mirror https://hostname/ghuser/repo.git
# Pushes the mirror to the new GitHub Enterprise repository
  1. Delete temporary dir:
$ cd ..
$ rm -rf repo.git

Notes

1 Due to the high I/O, it is recommended to rewrite the history into a temporary directory off-disk with the -d option, e.g. on tmpfs.

Upvotes: 0

Related Questions