Force LF eol in git repo and working copy

Question

I have a git repository hosted on github. Many of the files were initially developed on Windows, and I wasn't too careful about line endings. When I performed the initial commit, I also didn't have any git configuration in place to enforce correct line endings. The upshot is that I have a number of files with CRLF line endings in my github repository.

I'm now developing partially on Linux, and I'd like to clean up the line endings. How can I ensure the files are stored correctly with LF on github, and have LF in my working copy?

I've set up a .gitattributes file containing text eol=LF; is that correct? With that committed and pushed, can I just rm my local repo and re-clone from github to get the desired effect?

Answer 1

Starting with Git 2.10 (released 2016-09-03), it is not necessary to enumerate each text file separately. Git 2.10 fixed the behavior of text=auto together with eol=lf. Source.

.gitattributes file in the root of your Git repository:

* text=auto eol=lf

Add and commit it.

Afterwards, you can do the following two steps to normalize all files:

git rm --cached -r .  # Remove every file from git's index.
git reset --hard      # Rewrite git's index to pick up all the new line endings.

Source: Answer by kenorb.

Warning

In some cases, files might be corrupted. This is a very rare case and none of the reporters made a reproducer available.

Git checks the first 8000 bytes of the file for a NUL "Character" to determine if the file is binary. Some binary file formats are more likely to contain 8000 bytes without 8 consecutive zero bits and should be marked as binary in the .gitattribute file to avoid that issue.

Based on this answer and this answer. Kudos to DecimalTurn for the explanation

Answer 2

I was cloning the Chromium depot_tools to my mac and all files of the work copy were ended with CRLF. I found this script which fixed my problem.

cd <your repo>
# config the local repo to use LF
git config core.eol lf
git config core.autocrlf input

# Copy files from the index to the working tree
git checkout-index --force --all

# If above line doesn't work, delete all cached files and reset.
git rm --cached -r .
git reset --hard

Answer 3

To force LF line endings for all text files, you can create .gitattributes file in top-level of your repository with the following lines (change as desired):

# Ensure all C and PHP files use LF.
*.c         eol=lf
*.php       eol=lf

which ensures that all files that Git considers to be text files have normalized (LF) line endings in the repository (normally core.eol configuration controls which one do you have by default).

Based on the new attribute settings, any text files containing CRLFs should be normalized by Git. If this won't happen automatically, you can refresh a repository manually after changing line endings, so you can re-scan and commit the working directory by the following steps (given clean working directory):

$ echo "* text=auto" >> .gitattributes
$ rm .git/index     # Remove the index to force Git to
$ git reset         # re-scan the working directory
$ git status        # Show files that will be normalized
$ git add -u
$ git add .gitattributes
$ git commit -m "Introduce end-of-line normalization"

or as per GitHub docs:

git add . -u
git commit -m "Saving files before refreshing line endings"
git rm --cached -r . # Remove every file from Git's index.
git reset --hard # Rewrite the Git index to pick up all the new line endings.
git add . # Add all your changed files back, and prepare them for a commit.
git commit -m "Normalize all the line endings" # Commit the changes to your repository.

See also: @Charles Bailey post.

In addition, if you would like to exclude any files to not being treated as a text, unset their text attribute, e.g.

manual.pdf      -text

Or mark it explicitly as binary:

# Denote all files that are truly binary and should not be modified.
*.png binary
*.jpg binary

---

To see some more advanced git normalization file, check .gitattributes at Drupal core:

# Drupal git normalization
# @see https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html
# @see https://www.drupal.org/node/1542048

# Normally these settings would be done with macro attributes for improved
# readability and easier maintenance. However macros can only be defined at the
# repository root directory. Drupal avoids making any assumptions about where it
# is installed.

# Define text file attributes.
# - Treat them as text.
# - Ensure no CRLF line-endings, neither on checkout nor on checkin.
# - Detect whitespace errors.
#   - Exposed by default in `git diff --color` on the CLI.
#   - Validate with `git diff --check`.
#   - Deny applying with `git apply --whitespace=error-all`.
#   - Fix automatically with `git apply --whitespace=fix`.

*.config  text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.css     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.dist    text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.engine  text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.html    text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=html
*.inc     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.install text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.js      text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.json    text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.lock    text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.map     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.md      text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.module  text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.php     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.po      text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.profile text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.script  text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.sh      text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.sql     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.svg     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.theme   text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2 diff=php
*.twig    text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.txt     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.xml     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2
*.yml     text eol=lf whitespace=blank-at-eol,-blank-at-eof,-space-before-tab,tab-in-indent,tabwidth=2

# Define binary file attributes.
# - Do not treat them as text.
# - Include binary diff in patches instead of "binary files differ."
*.eot     -text diff
*.exe     -text diff
*.gif     -text diff
*.gz      -text diff
*.ico     -text diff
*.jpeg    -text diff
*.jpg     -text diff
*.otf     -text diff
*.phar    -text diff
*.png     -text diff
*.svgz    -text diff
*.ttf     -text diff
*.woff    -text diff
*.woff2   -text diff

---

See also:

• Dealing with line endings at GitHub
• When using vagrant: Windows CRLF to Unix LF Issues

Answer 4

Without a bit of information about what files are in your repository (pure source code, images, executables, ...), it's a bit hard to answer the question :)

Beside this, I'll consider that you're willing to default to LF as line endings in your working directory because you're willing to make sure that text files have LF line endings in your .git repository wether you work on Windows or Linux. Indeed better safe than sorry....

However, there's a better alternative: Benefit from LF line endings in your Linux workdir, CRLF line endings in your Windows workdir AND LF line endings in your repository.

As you're partially working on Linux and Windows, make sure core.eol is set to native and core.autocrlf is set to true.

Then, replace the content of your .gitattributes file with the following

* text=auto

This will let Git handle the automagic line endings conversion for you, on commits and checkouts. Binary files won't be altered, files detected as being text files will see the line endings converted on the fly.

However, as you know the content of your repository, you may give Git a hand and help him detect text files from binary files.

Provided you work on a C based image processing project, replace the content of your .gitattributes file with the following

* text=auto
*.txt text
*.c text
*.h text
*.jpg binary

This will make sure files which extension is c, h, or txt will be stored with LF line endings in your repo and will have native line endings in the working directory. Jpeg files won't be touched. All of the others will be benefit from the same automagic filtering as seen above.

In order to get a get a deeper understanding of the inner details of all this, I'd suggest you to dive into this very good post "Mind the end of your line" from Tim Clem, a Githubber.

As a real world example, you can also peek at this commit where those changes to a .gitattributes file are demonstrated.

UPDATE to the answer considering the following comment

I actually don't want CRLF in my Windows directories, because my Linux environment is actually a VirtualBox sharing the Windows directory

Makes sense. Thanks for the clarification. In this specific context, the .gitattributes file by itself won't be enough.

Run the following commands against your repository

$ git config core.eol lf
$ git config core.autocrlf input

As your repository is shared between your Linux and Windows environment, this will update the local config file for both environment. core.eol will make sure text files bear LF line endings on checkouts. core.autocrlf will ensure potential CRLF in text files (resulting from a copy/paste operation for instance) will be converted to LF in your repository.

Optionally, you can help Git distinguish what is a text file by creating a .gitattributes file containing something similar to the following:

# Autodetect text files
* text=auto

# ...Unless the name matches the following
# overriding patterns

# Definitively text files 
*.txt text
*.c text
*.h text

# Ensure those won't be messed up with
*.jpg binary
*.data binary

If you decided to create a .gitattributes file, commit it.

Lastly, ensure git status mentions "nothing to commit (working directory clean)", then perform the following operation

$ git checkout-index --force --all

This will recreate your files in your working directory, taking into account your config changes and the .gitattributes file and replacing any potential overlooked CRLF in your text files.

Once this is done, every text file in your working directory WILL bear LF line endings and git status should still consider the workdir as clean.