CODEWITHSUNDEEP
pythonpython-3.xunicodeutf-8character-encoding
Leonard Saers
Leonard Saers

Reputation: 669

Convert utf-8 unicode sequence to utf-8 chars in Python 3

I'm reading data from an aws s3 bucket which happens to have unicode chars escaped with double backslashes.

The double backslashes makes the unicode sequence parsed as a series of utf-8 characters instead of the character which the unicode represents.

The example illustrates the situation.

>>> s1="1+1\\u003d2"
>>> print(s1)
1+1\u003d2

The actual unicode sequence would in this case an equal sign.

>>> s2="1+1\u003d2"
>>> print(s2)
1+1=2

Is there a way to convert the sequence of utf-8 character in the first example so that the string represented by s1 is parsed with it's unicode sequence as the actual utf-8 sign that it represents?

Upvotes: 1

Views: 1396

Answers (2)

Leonard Saers
Leonard Saers

Reputation: 669

I'm also adding a variant of juanpa.arrivillaga solution which also handles surrogate escape.

>>> import codecs
>>> s1="A surrogate sequence \\ud808\\udf45"
>>> print(codecs.decode(s1, encoding='unicode_escape'))
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 21-22: surrogates not allowed
>>> print(codecs.decode(s1,encoding='unicode_escape',errors='surrogateescape').encode('utf-16', 'surrogatepass').decode('utf-16'))
A surrogate sequence 𒍅

Upvotes: 0

juanpa.arrivillaga
juanpa.arrivillaga

Reputation: 95948

I believe that the codecs module provides this utility:

>>> import codecs
>>> codecs.decode("1+1\\u003d2", encoding='unicode_escape')
'1+1=2'

This probably points to a larger problem, though. How do these strings come to be in the first place?

Note, if this is being extracted from a valid JSON string (in this case it would be missing the quotes), you could simply use:

>>> import json
>>> json.loads('"1+1\\u003d2"')
'1+1=2'

Upvotes: 5

Related Questions