How can I get rid of special characters in a text file? (Unix-like)

Question

I've got a source code file that started as a copy of some sample code from a webpage. It was created and edited under Windows and compiled without any problems.

But under Macs, I get a load of obscure errors, like:

../MyProgram.cpp:1: error: stray '\255' in program
../MyProgram.cpp:1: error: stray '\254' in program
../MyProgram.cpp:1: error: stray '#' in program

../MyProgram.cpp:3:4: error: invalid preprocessing directive #i
../MyProgram.cpp:5:4: error: invalid preprocessing directive #i
../MyProgram.cpp:7:4: error: invalid preprocessing directive #i
../MyProgram.cpp:23: error: missing terminating ' character
../MyProgram.cpp:369:6: error: invalid preprocessing directive #i
../MyProgram.cpp:371:8: error: invalid preprocessing directive #i
../MyProgram.cpp:375:8: error: invalid preprocessing directive #e
../MyProgram.cpp:381:8: error: invalid preprocessing directive #e
../MyProgram.cpp:383:6: error: invalid preprocessing directive #e
../MyProgram.cpp:385:8: error: invalid preprocessing directive #i
../MyProgram.cpp:389:8: error: invalid preprocessing directive #e

../MyProgram.cpp:1: error: 'i' does not name a type
../MyProgram.cpp:53: error: 'V' does not name a type
../MyProgram.cpp:75: error: 'v' does not name a type
../MyProgram.cpp:157: error: 'l' does not name a type
../MyProgram.cpp:169: error: 'l' does not name a type
../MyProgram.cpp:187: error: 'i' does not name a type
../MyProgram.cpp:197: error: 'v' does not name a type

It looks like the problem is with some special characters.

How can I strip them off with a Unix-like command line?

Answer 1

It looks to me as if the file was saved as UTF-16. Opening it in a text editor and re-encoding to UTF-8 should, with some luck, fix the problem.

Answer 2

Originally, I was just going say how to remove the \255 and \254 characters, but I agree with the comments. It's in Unicode.

Try to convert it with iconv:

iconv -f iso-8859-1 -t utf-8 infile > outfile

iso-8859-1 is just a guess.