CODEWITHSUNDEEP
pythonflaskpython-requests
Meeron
Meeron

Reputation: 13

Sending metadata along with a dataframe using Requests POST request

I'm using pythons Requests to send a Pandas Dataframe to a Flask server. The dataframe has about 2 million rows and 16 columns. I want to send a config dictionary along with the dataframe, as metadata. At the moment I am able to send the dataframe as a JSON file, however, I can't find any way to attach the metadata in the same post request.

Here's my code:

Client side:

 # Post request containing 1. The dataset (pandas df) 2. The metadata (dict)

 dataset = dataset.to_json(orient='split')
 metadata = {'dataset ID': "makis", 'date start': "1", 'date end': "2"}

 url = "http://localhost:8081/upload_dataset"
 r = requests.post(url, data=dataset)
 return r.text

Server side:

@app.route("/upload_dataset", methods=['POST'])

def upload_dataset():
    from werkzeug.datastructures import FileStorage

    payload = request.stream
    dataset = pd.read_json(payload, typ='frame', orient='split')
    FileStorage(payload).save('dataset.csv')

    return 'File Uploaded & Standing by', 200

Upvotes: 1

Views: 6406

Answers (1)

Diane M
Diane M

Reputation: 1512

Once serialized to json, your dataset is plain text. To add more parameters from there, your only options are embed your payload along with metadata in post parameters, resulting in url-encoding the json. Or embed your payload in a top-level json post, thus double-encode in json.

You would gain in clarity and maybe performance if you left json encoding job to requests instead. In this way you could add data and still encode/decode only once.

Example


dataset = dataset.to_dict(orient='list')
post_data = {'dataset ID': "makis", 'date start': "1", 'date end': "2", 'payload': dataset}
url = "http://localhost:8081/upload_dataset"
r = requests.post(url, json=post_data)

Server side:

@app.route("/upload_dataset", methods=['POST'])
def upload_dataset():
    post_data = request.get_json()
    ## Use of meta data keys e.g. post_data['date start']
    dataset = pd.from_dict(post_data['payload'], orient='columns')

Upvotes: 2

Related Questions