CODEWITHSUNDEEP
utf-8greputf-16
Light Yagami
Light Yagami

Reputation: 1043

Why doesn't grep command work on text files with UTF-16 LE encoding?

I want to save all the lines in a text file that start with certain string in another text file. So, I used this grep command to do that:

grep '^This' input.txt > output.txt

But the output file output.txt is empty though there are lot of lines in the file input.txt which start with the word 'this'. One of my mentor suggested that the file input.txt is in UTF-16 LE format and asked me to change it into UTF-8. Then the command worked well.

Why doesn't grep command work on files with UTF-16 LE format?

Upvotes: 4

Views: 3150

Answers (1)

deceze
deceze

Reputation: 522597

grep is not encoding aware. It doesn't search for "characters", it searches for bytes. Your console is sending UTF-8/ASCII encoded text (same in this case for the string "^This") to grep to search for. If the file contains UTF-16 encoded text, that won't match, since the byte representations are different.

Upvotes: 7

Related Questions