CODEWITHSUNDEEP
python-3.xunicodedecode
NealWalters
NealWalters

Reputation: 18187

Python decode from a specific string

Someone gave me this string: "Al Baţḩah" (it's probably an Arabic name), and asked me to translate it (and a list of other similar strings) in Python.

This website converts it to "Al Baţḩah" with the operation "UTF8 Decode". I'm trying to do the same in Python, but is what I tried and my results. Most of the examples took a series of unicode bytes. I tried the "detect" to see exactly what it is he was giving me. If it's already UTF8, then I'm not sure what that website is convert it to.

import chardet

byte_string = b"\x61\x62\x63"
decoded_string = byte_string.decode("utf8")
print(decoded_string)

sourceText = "Al Baţḩah"
sourceTextBytes = bytes(sourceText, 'utf-8')
print(chardet.detect(sourceTextBytes))
decoded_string2 = sourceTextBytes.decode("utf")
print("Result2=", decoded_string2)

Output of above:

abc
{'encoding': 'utf-8', 'confidence': 0.9690625, 'language': ''}
Result2= Al Baţḩah

The output is same as the input. I've tried ascic, utf8, etc... as the parms for the decode statement.

Part 2 - Here's another weird one that the solution below didn't work for (these are subdivison names from an ISO document a colleague purchased.)

Gədəbəy

Upvotes: 0

Views: 128

Answers (1)

anthony sottile
anthony sottile

Reputation: 69884

looks like a classic case of mojibake -- in this case it's interpreted using latin1 when it should be UTF-8:

>>> "Al Baţḩah".encode('latin1')
b'Al Ba\xc5\xa3\xe1\xb8\xa9ah'
>>> "Al Baţḩah".encode('latin1').decode('UTF-8')
'Al Baţḩah'

Code for those that want to copy/paste into a program instead of command line:

source_text = "Al Baţḩah"
print("source_text=", source_text)
encoded_source_text = source_text.encode('latin1')
decoded_text = encoded_source_text.decode('UTF-8')
print("decoded_text=", decoded_text)

Upvotes: 1

Related Questions