CODEWITHSUNDEEP
pythonjsonpython-3.xpython-requestsweb.py
Ludo
Ludo

Reputation: 2527

python3 - How to overcome max url limit with python requests

I have two python apps running on separate ports using web.py. I am trying to send JSON strings in the range of 30,000-40,000 characters long from one app to another. The JSON contains all the information necessary to generate a powerpoint report. I tried enabling this communication using requests as such:

import requests
template = <long JSON string>     
url = 'http://0.0.0.0:6060/api/getPpt?template={}'.format(template)
resp= requests.get(url).text

I notice that on the receiving end the json has been truncated to 803 characters long and so when it decodes the JSON I get:

json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 780 (char 779)

I assume this has to with a limitation on how long a URL request can be, from either web.py or requests or this is a standardised thing. Is there a way around this or do I need to find another way of enabling communication between these two python apps. If sending such long JSONs via http isn't possible could you please suggest alternatives. Thanks!

Upvotes: 2

Views: 12194

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1122132

Do not put that much data into a URL. Most browsers limit the total length of a URL (including the query string) to about 2000 characters, servers to about 8000.

See What is the maximum length of a URL in different browsers?, which quotes the HTTP/1.1 standard, RFC7230:

Various ad hoc limitations on request-line length are found in practice. It is RECOMMENDED that all HTTP senders and recipients support, at a minimum, request-line lengths of 8000 octets.

You need to send that much data in the request body instead. Use POST or PUT as the method.

The requests library itself does not place any limits on the URL length; it sends the URL to the server without truncating it. It is your server that has truncated it here, instead of giving you a 414 URI Too Long status code.

Upvotes: 5

Related Questions