CODEWITHSUNDEEP
pythonjsonpandascurlflask
B_Miner
B_Miner

Reputation: 1832

Flask Request passing pandas to_json versus curl or Postman

I am calling a flask app that is hosted with AWS Elastic Beanstalk. The app takes in json, creates a pandas data frame from it, does some other processing and passes data into a model. The predictions are returned.

If I run the following, the code executes and returns back the proper response.

import requests
v=pandas_df.iloc[0:5].to_json(orient='records')
headers = {'content-type': 'application/json'}
r = requests.post('http://XXXXXXXXX.us-east-1.elasticbeanstalk.com/', json=v, headers = headers)
r.json() #the predictions

However I can not get the same result using curl or postman.

For curl I have tried to send the json object created by pandas:

pandas_df.iloc[0:5].to_json(orient='records',path_or_buf=jsonexp.json)

curl -X POST "http://XXXXXXXXX.us-east-1.elasticbeanstalk.com/" -H "Content-Type: application/json" -d "jsonexp.json"

but the result is 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>

I have tried passing the string version of v, for the sample of 5 records above 

curl -X POST "http://XXXXXXXXX.us-east-1.elasticbeanstalk.com/" -H "Content-Type: application/json" -d '[{"field1":14239302,"field2":29....}]'

but it returns 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

How can I get the same result request through curl?

ADD

Here is a reproducible example. The issue seems to be in how the json is read for creation of a pandas data frame.

Here is the flask app:

from flask import Flask, jsonify, request, render_template, make_response
import pandas as pd


application = Flask(__name__)


@application.route('/', methods=['GET','POST'])
def apicall():
    if request.method == 'POST':

        f = request.get_json()
        print(f)
        print(type(f))
        f=str(f)
        print(type(f))
        df=pd.read_json(f,orient='records')
        print(df)

        return(jsonify(df.to_json(orient='records')))

if __name__ == '__main__':

    application.run()

This works perfectly:

r=requests.post('http://127.0.0.1:5000/', json='[{"field1":14239302,"field2":29.90}]', headers = {'content-type': 'application/json'})
r.json()

127.0.0.1 - - [13/Sep/2019 15:28:16] "POST / HTTP/1.1" 500 -
[{"field1":14239302,"field2":29.90}]
<class 'str'>
<class 'str'>
     field1  field2
0  14239302    29.9
127.0.0.1 - - [13/Sep/2019 15:34:51] "POST / HTTP/1.1" 200 -

This fails:

!curl -d '[{"field1":14239302,"field2":29.90}]' -H "Content-Type: application/json" -X POST 'http://127.0.0.1:5000/'

with the following

127.0.0.1 - - [13/Sep/2019 15:28:13] "POST / HTTP/1.1" 200 -
{'field1': 14239302, 'field2': 29.9}
<class 'list'>
<class 'str'>
[2019-09-13 15:28:16,851] ERROR in app: Exception on / [POST]
Traceback (most recent call last):
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "application.py", line 17, in apicall
    df=pd.read_json(f,orient='records')
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 592, in read_json
    result = json_reader.read()
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 717, in read
    obj = self._get_object_parser(self.data)
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 739, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 849, in parse
    self._parse_no_numpy()
  File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1116, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value
127.0.0.1 - - [13/Sep/2019 15:28:16] "POST / HTTP/1.1" 500 -

It seems like the issue is with the fact that the input to the flask app is a string for requests and a list for curl?

ADD BASED ON KEVIN's ANSWER:

I tried this function:

from flask import Flask, jsonify, request, render_template, make_response
import pandas as pd


application = Flask(__name__)


@application.route('/', methods=['GET','POST'])
def apicall():
    if request.method == 'POST':

        f = request.get_json(force=True)
        print(f)
        print(type(f))

        df=pd.read_json(f,orient='records')

        return(jsonify(df.to_json(orient='records')))

if __name__ == '__main__':

    application.run()

and this input:

s=json.dumps([{'field1':14239302,'field2':29.90}])

!curl  -d s -H "Content-Type:application/json" -X POST 'http://127.0.0.1:5000/'

But still receive an error:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>

Upvotes: 4

Views: 1556

Answers (2)

stasiekz
stasiekz

Reputation: 1863

I guess you are using ipython shell. s variable is not rendered unless you put it in curly brackets. Try

s=json.dumps([{'field1':14239302,'field2':29.90}]) 
!curl -d '{s}' -H "Content-Type:application/json" -X POST 'http://127.0.0.1:5000/'

s is not valid JSON hence 400 Error.

However the problem's cause is different.

In fact Flask get_json returns str when request is made by requests package or list is returned when using curl which causes 500 error. This is because requests module escapes quote character and request's body looks this way: b'"[{\\"field1\\":14239302,\\"field2\\":29.90}]"' while body from curl is b'[{"field1": 14239302, "field2": 29.9}]'.

To make it work with curl you will need to escape " quote characters inside dict and put it in quotes like so:

curl -H "Content-Type: application/json" -d '"[{\"field1\":14239302,\"field2\":29.9}]"' -X POST 'http://127.0.0.1:5000/'`

Upvotes: 2

Kevin Welch
Kevin Welch

Reputation: 1508

pd.read_json requires a valid JSON string.

You are trying to use string casting to convert to JSON. str(f) is not a reliable way to convert to JSON.

The reason it works with requests is because you got lucky... you can see that it prints valid JSON: [{"field1":14239302,"field2":29.90}]

However when you run curl, you do not have valid JSON due to the single quotes.

{'field1': 14239302, 'field2': 29.9}

Try using the json module to convert Python objects to a valid JSON string.

import json

json_string = json.dumps([{'field1': 14239302, 'field2': 29.9}])
print(json_string)
# '[{"field1":14239302,"field2":29.90}]'
# see how it prints your json_string with double quotes wrapping the keys

Upvotes: 0

Related Questions