CODEWITHSUNDEEP
pythonunicodeencodingutf-8
leopik
leopik

Reputation: 2351

Converting unicode string to utf-8

Firstly, I am aware that there are tons of questions regarding en/de-coding of strings in Python 2.x, but I can't seem to find a solution to this problem.

I have a unicode string, that contains letter č which is represented as \u00c4\u008d

If in Python console I write

>>> a = u"\u00c4\u008d"
>>> print a

I get two strange characters printed out instead of č, probably because the actual encoding of that string is supposed to be UTF-8. Therefore I try to use .decode("utf-8") but for this I get the standard UnicodeEncodeError.

Do you know how I can make Python print that string as č in the console?

Upvotes: 1

Views: 9516

Answers (2)

Mark Tolonen
Mark Tolonen

Reputation: 177461

č is not represented by u'\u00c4\u008d'. Those two hex values are the UTF-8-encoded values, so should be written in a byte string as '\xc4\x8d'. Example:

>>> s = '\xc4\x8d'
>>> s.decode('utf8')
u'\u010d'
>>> print(s.decode('utf8'))
č

Caveat: Your terminal must be configured with an encoding that supports the character to print correctly, or you will see a UnicodeEncodeError.

If for some reason you have a mis-decoded Unicode string, you can take advantage of the fact that the first 256 code points of Unicode correlate to the latin1 encoding and fix it:

>>> s = u'\u00c4\u008d'
>>> s.encode('latin1')
'\xc4\x8d'
>>> s.encode('latin1').decode('utf8')
u'\u010d'
>>> print(s.encode('latin1').decode('utf8'))
č

If you have a mis-decoded Unicode string, you should show the file you have or the code that read it and solve that problem.

Upvotes: 3

leopik
leopik

Reputation: 2351

After fighting with python for over an hour, I decided to look for a solution in another language. This is how my goal can be achieved in C#:

var s = "\u00c4\u008d";
var newS = Encoding.UTF8.GetString(Encoding.Default.GetBytes(s));
File.WriteAllText(@"D:\tmp\test.txt", newS, Encoding.UTF8);

Finally! The file now contains č.

I therefore got inspired by this approach in C# and managed to come up with the following (seemingly) equivalent solution in Python:

>>> s = u"\u00c4\u008d"
>>> arr = bytearray(map(ord, s))
>>> print arr.decode("utf-8")
č

I'm not sure how good this solution is but it seems to work in my case.

Upvotes: 0

Related Questions