CODEWITHSUNDEEP
pythonjsonpandas
ehacinom
ehacinom

Reputation: 8894

ValueError when using pandas.read_json

I made a 250MB json file that should look like this:

[ {"A":"uniquevalue0", "B":[1,2,3]}, 
  {"A":"uniquevalue1", "B":[1]}, 
  {"A":"uniquevalue2", "B":[1,2,3,4]} ]

where the "B" value can be variable len >= 1. This says I have valid JSON.

I call

df = pandas.read_json('ut1.json', orient = 'records', dtype={"A":str, "B":list})

Here is the documentation. When reading into a pandas dataframe, I get the following traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../pandas/io/json.py", line 198, in read_json     
    date_unit).parse()
  File "/.../pandas/io/json.py", line 266, in parse 
    self._parse_no_numpy()
  File "/.../pandas/io/json.py", line 496, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Unexpected character found when decoding 'true'

Can't think of what is going wrong. The python file that is throwing the error is not that helpful.

Upvotes: 29

Views: 82348

Answers (10)

theartist33
theartist33

Reputation: 482

  1. First make sure your file is there, now ofcourse it is but what worked for me is deleting the file and recopying to the location and be cautious while renaming it and then read the file as :

    df_test = pd.read_json('test_file.json',lines=True)

Note that I have had encountered all different errors talked in this particular thread and finally this solution work which I have explained i.e. recopying the file without renaming it and then reading the file ( also making sure my code and file shares the same dir if I am not providing absolute path while reading the file).

Upvotes: 0

Preshma Linet Pereira
Preshma Linet Pereira

Reputation: 350

try this

df = pd.read_json('file.jsonl', lines=True) #if its a jsonl file

Upvotes: 1

KrisG
KrisG

Reputation: 149

For me, the issue was that the file has a UTF-8 BOM character at the beginning. Using the following encoding solved the problem:

df = pd.read_json(r'C:\temp\foo.jsonl', lines=True, encoding='utf-8-sig')

Upvotes: 2

Nishad Kukarni
Nishad Kukarni

Reputation: 1

I had the same error when running the code on linux. I realised the file name given to read_csv had .JSON instead of .json. Changing it to lower case worked for me.

Upvotes: 0

Arun Krishnan
Arun Krishnan

Reputation: 271

I had the same issue and then realized that the issue was while copying and pasting text from the browser to my file. It introduced carriage returns so that each line, for a given key was broken up into multiple rows. THAT was the issue. hope this helps someone!

Upvotes: 4

flome
flome

Reputation: 1

Posting this because the answers above did not match my problem with this error: It just occurred for me when reading a very long string document I thought would be valid json format but contained nan as exported from python to string while it should be "null" for valid json. If the document was not created using a json package it might have faulty values indicated by this error message.

Upvotes: 0

typhon04
typhon04

Reputation: 2624

In my case, the path was wrong.

Make sure you check your current working directory, by placing this just before the pandas.read_json:

import os
print(os.getcwd())

Upvotes: 9

imgg
imgg

Reputation: 468

After tried @learn2day's answer, I still cannot get a good result from there, but I do try the following code and everything works for me. (PS: I'm opening a JSON file where Chinese characters were UTF-8 characters appeared - Chinese characters)

pandas.read_json(open("ut1.json", "r", encoding="utf8"))

The encoding="utf8" is the key part of this code.

Upvotes: 3

user3614003
user3614003

Reputation: 11

I was getting the "Value Error: Expected object or value" today while calling pandas.read_json('my_file.json'). I have ran this code with the same file earlier, so was very worried to see it is not working today. Later, I found for some reason, the json file was not there in the same dir. Then, I downloaded the file from git by right clicking the file link. That was a bad idea :(. I guess the json file was not encoded properly, so I kept getting the same error even when the json file was there in the same dir. At the end, I deleted the json file, cloned the original git repo to get the json file and put it in the same dir again. Then, this pandas.read_json did work. So, first of all please make sure the json file exists in the proper dir and then make sure, the file is not corrupted.

Upvotes: 1

learn2day
learn2day

Reputation: 1716

I had the same error message, and I solved it by using an absolute path.

import os
basePath = os.path.dirname(os.path.abspath(__file__))
df = pandas.read_json(basePath + '/ut1.json', orient = 'records', dtype={"A":str, "B":list})

That worked for me!

Upvotes: 27

Related Questions