CODEWITHSUNDEEP
pythonencodingutf-8
Ash
Ash

Reputation: 785

Convert escaped utf-8 string to utf in python 3

I have a py3 string that includes escaped utf-8 sequencies, such as "Company\\ffffffc2\\ffffffae", which I would like to convert to the correct utf 8 string (which would in the example be "Company®", since the escaped sequence is c2 ae). I've tried

print (bytes("Company\\\\ffffffc2\\\\ffffffae".replace(
    "\\\\ffffff", "\\x"), "ascii").decode("utf-8"))

result: Company\xc2\xae

 print (bytes("Company\\\\ffffffc2\\\\ffffffae".replace (
     "\\\\ffffff", "\\x"), "ascii").decode("unicode_escape"))

result: Company®

(wrong, since chracters are treated separately, but they should be treated together.

If I do

print (b"Company\xc2\xae".decode("utf-8"))

It gives the correct result. Company®

How can i achieve that programmatically (i.e. starting from a py3 str)

Upvotes: 1

Views: 669

Answers (1)

VPfB
VPfB

Reputation: 17292

A simple solution is:

import ast

test_in = "Company\\\\ffffffc2\\\\ffffffae"
test_out = ast.literal_eval("b'''" + test_in.replace('\\\\ffffff','\\x') + "'''").decode('utf-8')
print(test_out)

However it will fail if there is a triple quote ''' in the input string itself.

Following code does not have this problem, but it is not as simple as the first one.

In the first step the string is split on a regular expression. The odd items are ascii parts, e.g. "Company"; each even item corresponds to one escaped utf8 code, e.g. "\\\\ffffffc2". Each substring is converted to bytes according to its meaning in the input string. Finally all parts are joined together and decoded from bytes to a string.

import re

REGEXP = re.compile(r'(\\\\ffffff[0-9a-f]{2})', flags=re.I)

def convert(estr):
    def split(estr):
        for i, substr in enumerate(REGEXP.split(estr)):
            if i % 2:
                yield bytes.fromhex(substr[-2:])
            elif substr:
                yield bytes(substr, 'ascii')
    return b''.join(split(estr)).decode('utf-8')


test_in = "Company\\\\ffffffc2\\\\ffffffae"
print(convert(test_in))

The code could be optimized. Ascii parts do not need encode/decode and consecutive hex codes should be concatenated.

Upvotes: 1

Related Questions