CODEWITHSUNDEEP
pythonjsonpython-3.xpandas
Alain Michael Janith Schroter
Alain Michael Janith Schroter

Reputation: 693

appending json files in python

I am trying to append some json files in python. I have the following code. It seems right. However, I am getting an error.

The code is as follows.

import pandas as pd

df1=pd.DataFrame()
for i in range(0,49):
    df = pd.read_json ('/media/michael/extHDD/Kaggle/DeepFAke/DF_all/metadata{}.json'.format(i))
    df1.append(df.T)

The error is as follows.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-76-ddb355627155> in <module>
      3 df1=pd.DataFrame()
      4 for i in range(0,49):
----> 5     df = pd.read_json ('/media/michael/extHDD/Kaggle/DeepFAke/DF_all/metadata{}.json'.format(i))
      6     df1.append(df.T)

~/myenv/lib/python3.5/site-packages/pandas/io/json/_json.py in read_json(path_or_buf, orient, typ, dtype, convert_axes, convert_dates, keep_default_dates, numpy, precise_float, date_unit, encoding, lines, chunksize, compression)
    590         return json_reader
    591 
--> 592     result = json_reader.read()
    593     if should_close:
    594         try:

~/myenv/lib/python3.5/site-packages/pandas/io/json/_json.py in read(self)
    715             obj = self._get_object_parser(self._combine_lines(data.split("\n")))
    716         else:
--> 717             obj = self._get_object_parser(self.data)
    718         self.close()
    719         return obj

~/myenv/lib/python3.5/site-packages/pandas/io/json/_json.py in _get_object_parser(self, json)
    737         obj = None
    738         if typ == "frame":
--> 739             obj = FrameParser(json, **kwargs).parse()
    740 
    741         if typ == "series" or obj is None:

~/myenv/lib/python3.5/site-packages/pandas/io/json/_json.py in parse(self)
    847 
    848         else:
--> 849             self._parse_no_numpy()
    850 
    851         if self.obj is None:

~/myenv/lib/python3.5/site-packages/pandas/io/json/_json.py in _parse_no_numpy(self)
   1091         if orient == "columns":
   1092             self.obj = DataFrame(
-> 1093                 loads(json, precise_float=self.precise_float), dtype=None
   1094             )
   1095         elif orient == "split":

ValueError: Expected object or value

The code works when I do it for each file individually. Would anyone be able to help me regarding this.

Thanks & Best Regards

Michael

Upvotes: 0

Views: 79

Answers (1)

Serge Ballesta
Serge Ballesta

Reputation: 148890

The error occurs on a df = pd.read_json (...) line. It is likely that one of the file is non existent or incorrect. My advice is to use a try catch to identify it:

for i in range(0,49):
    try:
        df = pd.read_json ('/media/michael/extHDD/Kaggle/DeepFAke/DF_all/metadata{}.json'.format(i))
    except:
        print('Error on iteration', i, ', file',
              '/media/michael/extHDD/Kaggle/DeepFAke/DF_all/metadata{}.json'.format(i))
        raise
df1.append(df.T)

Catching any exception is normally bad practice because it can hide truely abnormal conditions like an IO or memory error. That is the reason why I re-raise the original exception in above code.

Upvotes: 1

Related Questions