CODEWITHSUNDEEP
bashunicodegrep
Jonas Stein
Jonas Stein

Reputation: 7043

How to search for non-ASCII characters with bash tools?

I have a large text file that contains a few unicode characters that make LaTeX crash. How can I find non-ASCII characters in a file with sed, and the like in a Linux bash?

Upvotes: 32

Views: 26265

Answers (2)

pixelbeat
pixelbeat

Reputation: 31718

Try:

nonascii() { LANG=C grep --color=always '[^ -~]\+'; }

Which can be used like:

printf 'ŨTF8\n' | nonascii

Within [] ^ means "not". So [^ -~] means characters not between space and ~. So excluding control chars, this matches non ASCII characters, and is a more portable though slightly less accurate version of [^\x00-\x7f] below. The \+ means 1 or more and will get multibye characters to have a color shown around the complete character(s), rather than interspersed in each byte, thus corrupting the multibyte sequence

Upvotes: 39

kev
kev

Reputation: 161674

Try this command:

grep -P '[^\x00-\x7f]' file

Upvotes: 32

Related Questions