Why does git treat this text file as binary and fail to provide meaningful diff?

Question

Doing an intro to git course on Pluralsight and git isn't cooperating with the diff section.

OS is Windows 8.1. I'm trying to diff two versions of a text file with no special characters. I created this text file in PowerShell via:

echo "Hello, world" > readme.txt

Regular diff command says that the binary files differ.

PS C:\GitTest> git diff HEAD~1..HEAD
diff --git a/readme.txt b/readme.txt
index 440580d..0d6852b 100644
Binary files a/readme.txt and b/readme.txt differ

When I force it with --text I get this output:

PS C:\GitTest> git diff --text HEAD~1..HEAD
diff --git a/readme.txt b/readme.txt
index 83f0e87..fef1216 100644
--- a/readme.txt
+++ b/readme.txt
@@ -1,2 +1,3 @@
-ÿþH-\ No newline at end of file
+ÿþH++\ No newline at end of file

Not sure why it thinks it is binary in the first place or why the above diff seems useless. The two versions of the file look like this:

HEAD~1:

Hello, Git!
Hello, again!
Hello, for the last time!

HEAD:

Hello, Git!
Hello, again!
Hello, for the last time!
Hello, again I hope this really is the last time...

My editor is using CRLF but I've got git set to core.autocrlf=true. I've ensured that the HEAD and HEAD~1 versions both have a newline at the end of the file.

I feel like I'm probably missing something simple - What is it? Any help is much appreciated.

Answer 1

Those first 2 characters in the diff (the y with dots on it and the thorn) are the Latin-1 interpretation of the bytes 0xFF and 0xFE. If those bytes were interpreted as UCS2/UTF16 instead, they'd be a little-endian Byte Order Mark. After those bytes you have the H of "Hello, world" and then probably a NUL which causes the forced-text-mode diff to give up.

So your text editor has saved the file in UTF16LE format, which is fairly common in Windows but rare in the unix world, where git comes from. That's why git is confused.

If you can tell your editor to save the file as UTF8 (or some other 8-bit encoding) then it will get along better with git. Or you might want to see this question about configuration options for git that make it work better with UTF16 files.