CODEWITHSUNDEEP
pythonjsonpython-3.x
min2bro
min2bro

Reputation: 4638

Python 3: JSON File Load with Non-ASCII Characters

just trying to load this JSON file(with non-ascii characters) as a python dictionary with Unicode encoding but still getting this error:

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 90: ordinal not in range(128)

JSON file content = "tooltip":{ "dxPivotGrid-sortRowBySummary": "Sort\"{0}\"byThisRow",}

import sys  
import json

data = []
with open('/Users/myvb/Desktop/Automation/pt-PT.json') as f:
    for line in f:
        data.append(json.loads(line.encode('utf-8','replace')))

Upvotes: 5

Views: 13462

Answers (4)

tdelaney
tdelaney

Reputation: 77347

You have several problems as near as I can tell. First, is the file encoding. When you open a file without specifying an encoding, the file is opened with whatever sys.getfilesystemencoding() is. Since that may vary (especially on Windows machines) its a good idea to explicitly use encoding="utf-8" for most json files. Because of your error message, I suspect that the file was opened with an ascii encoding.

Next, the file is decoded from utf-8 into python strings as it is read by the file system object. The utf-8 line has already been decoded to a string and is already ready for json to read. When you do line.encode('utf-8','replace'), you encode the line back into a bytes object which the json loads (that is, "load string") can't handle.

Finally, "tooltip":{ "navbar":"Operações de grupo"} isn't valid json, but it does look like one line of a pretty-printed json file containing a single json object. My guess is that you should read the entire file as 1 json object.

Putting it all together you get:

import json

with open('/Users/myvb/Desktop/Automation/pt-PT.json', encoding="utf-8") as f:
    data = json.load(f)

From its name, its possible that this file is encoded as a Windows Portugese code page. If so, the "cp860" encoding may work better.

Upvotes: 11

Magnun Leno
Magnun Leno

Reputation: 2738

You don't need to read each line. You have two options:

import sys  
import json

data = []
with open('/Users/myvb/Desktop/Automation/pt-PT.json') as f:
    data.append(json.load(f))

Or, you can load all lines and pass them to the json module:

import sys  
import json

data = []
with open('/Users/myvb/Desktop/Automation/pt-PT.json') as f:
    data.append(json.loads(''.join(f.readlines())))

Obviously, the first suggestion is the best.

Upvotes: 0

Jan Vlcinsky
Jan Vlcinsky

Reputation: 44112

Having a file with content similar to yours I can read the file in one simple shot:

>>> import json
>>> fname = "data.json"
>>> with open(fname) as f:
...     data = json.load(f)
...
>>> data
{'tooltip': {'navbar': 'Operações de grupo'}}

Upvotes: 0

Giovanni Rescia
Giovanni Rescia

Reputation: 590

I had the same problem, what worked for me was creating a regular expression, and parsing every line from the json file:

REGEXP = '[^A-Za-z0-9\'\:\.\;\-\?\!]+'
new_file_line = re.sub(REGEXP, ' ', old_file_line).strip()

Upvotes: 0

Related Questions