CODEWITHSUNDEEP
pythonpython-3.xencodingpython-unicode
Alex Hall
Alex Hall

Reputation: 61

Python decoding issue with Chinese characters

I'm using Python 3.5, and I'm trying to take a block of byte text that may or may not contain special Chinese characters and output it to a file. It works for entries that do not contain Chinese characters, but breaks when they do. The Chinese characters are always a person's name, and are always in addition to the English spelling of their name. The text is JSON formatted and needs to be decoded before I can load it. The decoding seems to go fine and doesn't give me any errors. When I try and write the decoded text to a file it gives me the following error message:

UnicodeEncodeError: 'charmap' codec can't encode characters in position 14-18: character maps to undefined

Here is an example of the raw data that I get before I do anything to it:

 b'  "isBulkRecipient": "false",\r\n      "name": "Name in, English \xef'
 b'\xab\x62\xb6\xe2\x15\x8a\x8b\x8a\xee\xab\x89\xcf\xbc\x8a",\r\n

Here is the code that I am using:

recipientData = json.loads(recipientContent.decode('utf-8', 'ignore'))
recipientName = recipientData['signers'][0]['name']
pprint(recipientName)
with open('envelope recipient list.csv', 'a', newline='') as fp:
    a = csv.writer(fp, delimiter=',')
    csvData = [[recipientName]]
    a.writerows(csvData)

The recipientContent is obtained from an API call. I do not need to have the Chinese characters in the output file. Any advice will be greatly appreciated!

Update:

I've been doing some manual workarounds for each entry that breaks, and came other entries that didn't contain Chinese special characters, but had them from other languages, and the broke the program as well. The special characters are only in the name field. So a name could be something like "Ałex" where it is a mixture of normal and special characters. Before i decode the string that contains this information i am able to print it out to the screen and it looks like this: b'name": "A\xc5ex",\r\n

But after i decode it into utf-8 it will give me an error if i try to output it. The error message is: UnicodeEncodeError: 'charmap' codec can't encode character 'u0142' in position 2- character maps to -undefined-

I looked up what \u0142 was and it is the ł special character.

Upvotes: 6

Views: 4823

Answers (2)

Alastair McCormack
Alastair McCormack

Reputation: 27704

The error you're getting is when you're writing to the file.

In Python 3.x, when you open() in text mode (the default) without specifying an encoding=, Python will use an encoding most suitable to your locale or language settings.

If you're on Windows, this will use the charmap codec to map to your language encoding.

Although you could just write bytes straight to a file, you're doing the right thing by decoding it first. As others have said, you should really decode using the encoding specified by the web server. You could also use Python Requests module, which does this for you. (You example doesn't decode as UTF-8, so I assume your example isn't correct)

To solve your immediate error, simply pass an encoding to open(), which supports the characters you have in your data. Unicode in UTF-8 encoding is the obvious choice. Therefore, you should change your code to read:

with open('envelope recipient list.csv', 'a', encoding='utf-8', newline='') as fp:

Upvotes: 2

Xavier Combelle
Xavier Combelle

Reputation: 11195

Warning: shotgun solution ahead

Assuming you just want to get rid of all foreign character in all your file ( that is they are not important for your future processing of all other fields), you can simply ignore all non ascii characters

recipientData = json.loads(recipientContent.decode('utf-8', 'ignore'))

by

recipientData = json.loads(recipientContent.decode('ascii', 'ignore'))

like this you remove all non ascii characters before future processing.

I called it shotgun solution because it might not work correctly under certain circumstances:

  1. Obviously if non ascii characters are needed to keep for future use
  2. If b'\' or b" characters appears for example from part of an utf-16 character.

Upvotes: 0

Related Questions