CODEWITHSUNDEEP
command-lineencodingscriptingcharacter-encoding
Marcin
Marcin

Reputation: 7994

Charset conversion from XXX to utf-8, command line

I have a bunch of text files that are encoded in ISO-8851-2 (have some polish characters). Is there a command line tool for linux/mac that I could run from a shell script to convert this to a saner utf-8?

Upvotes: 14

Views: 20838

Answers (3)

lhf
lhf

Reputation: 72422

Use iconv, for example like this:

iconv -f LATIN1 -t UTF-8 input.txt > output.txt

Some more information:

  • You may want to specify UTF-8//TRANSLIT instead of plain UTF-8. To quote the manpage:

    If the string //TRANSLIT is appended to to-encoding, characters being converted are transliterated when needed and possible. This means that when a character cannot be represented in the target character set, it can be approximated through one or several similar looking characters. Characters that are outside of the target character set and cannot be transliterated are replaced with a question mark (?) in the output.

  • For a full list of encoding codes accepted by iconv, execute iconv -l.

  • The example above makes use of shell redirection. Make sure you are not using a shell that mangles encodings on redirection – that is, do not use PowerShell for this.

Upvotes: 29

legoscia
legoscia

Reputation: 41658

recode latin2..utf8 myfile.txt

This will overwrite myfile.txt with the new version. You can also use recode without a filename as a pipe.

Upvotes: 10

Jonathan Leffler
Jonathan Leffler

Reputation: 755064

GNU 'libiconv' should be able to do the job.

Upvotes: 3

Related Questions