CODEWITHSUNDEEP
pythonjsonpandasrestflask
SSingh
SSingh

Reputation: 209

Multiple Json in Flask Response to convert to Pandas DataFrame

I have following JSON structure coming from a Flask Rest API. It is more than one JSON based on how many assets we query for and I am not able to convert it to Pandas dataframe.

from flask import Flask
import requests
import pandas as pd
import json

url = "http://localhost:5000/getpqdata"
random_cols = ['AAPL', 'MSFT']
JsonOutput = {'Assets': random_cols}

headers = {'Content-type': 'application/json'}
response = requests.post(url, json=JsonOutput, headers=headers)

rawdata = response.text

rawdata is coming as below: 

rawdata = '''[{"APPL": 1.067638}, {"AAPL": -1.996081}]
        [{"MSFT": 0.086638}, {"MSFT": -0.926081}]'''

data = json.loads(rawdata)

df = pd.DataFrame(data)
print(df)

It gives following error.

C:\Python36>python D:\Python\pyarrow\RestfulAPI\test.py
Traceback (most recent call last):
File "D:\Python\pyarrow\RestfulAPI\test.py", line 36, in <module>
data = json.loads(rawdata)
File "C:\Python36\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Python36\lib\json\decoder.py", line 342, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 13 (char 54)

Upvotes: 0

Views: 862

Answers (1)

noslenkwah
noslenkwah

Reputation: 1744

The problem you are having does not have anything to do with pandas but rather with the JSON decoding. json.loads(...) only supports one JSON object. Your rawdata has 2 JSON objects in it. Thus when it reaches the second line it tells you there is extra data. You can see a potential solution to that in this answer.

In short, you can do something like this:

def parse_json_stream(stream):
    decoder = json.JSONDecoder()
    while stream:
        obj, idx = decoder.raw_decode(stream)
        yield obj
        stream = stream[idx:].lstrip()

parsed_data = list(parse_json_stream(rawdata))
print(parsed_data)

[[{'APPL': 1.067638}, {'AAPL': -1.996081}], [{'MSFT': 0.086638}, {'MSFT': -0.926081}]]

As for converting it to a DataFrame, it depends on how you want to organize your data.

Upvotes: 1

Related Questions