How to handle "413: Request Entity Too Large" in python flask server

Question

I'm using Flask-uploads to upload files to my Flask server. The max size allowed is set by using flaskext.uploads.patch_request_class(app, 16 * 1024 * 1024).

My client application (A unit test) uses requests to post a file that is to large.

I can see that my server returnes a HTTP response with status 413: Request Entity Too Large. But the client raises an exception in the requests code

ConnectionError: HTTPConnectionPool(host='api.example.se', port=80): Max retries exceeded with url: /images (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)

My guess is that the server disconnect the receving socket and sends the reponse back to the client. But when the client gets a broken sending socket, it raises an exception and skips the response.

Questions:

• Are my guess about Flask-Uploads and requests correct?
• Does Flask-Uploads and request handle the 413 error correct?
• Should I expect that my client code gets back some html when the post are to large?

Update

Here is a simple example reproducing my problem.

server.py

from flask import Flask, request
app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 1024

@app.route('/post', methods=('POST',))
def view_post():
    return request.data

app.run(debug=True)

client.py

from tempfile import NamedTemporaryFile
import requests

def post(size):
    print "Post with size %s" % size,
    f = NamedTemporaryFile(delete=False, suffix=".jpg")
    for i in range(0, size):
        f.write("CoDe")
    f.close()

    # Post
    files = {'file': ("tempfile.jpg", open(f.name, 'rb'))}
    r = requests.post("http://127.0.0.1:5000/post", files=files)
    print "gives status code = %s" % r.status_code

post(16)
post(40845)
post(40846)

result from client

Post with size 16 gives status code = 200
Post with size 40845 gives status code = 413
Post with size 40846
Traceback (most recent call last):
  File "client.py", line 18, in <module>
    post(40846)
  File "client.py", line 13, in post
    r = requests.post("http://127.0.0.1:5000/post", files=files)
  File "/opt/python_env/renter/lib/python2.7/site-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/python_env/renter/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python_env/renter/lib/python2.7/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python_env/renter/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python_env/renter/lib/python2.7/site-packages/requests/adapters.py", line 354, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /post (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)

my versions

$ pip freeze
Flask==0.10.1
Flask-Mail==0.9.0
Flask-SQLAlchemy==1.0
Flask-Uploads==0.1.3
Jinja2==2.7.1
MarkupSafe==0.18
MySQL-python==1.2.4
Pillow==2.1.0
SQLAlchemy==0.8.2
Werkzeug==0.9.4
blinker==1.3
itsdangerous==0.23
passlib==1.6.1
python-dateutil==2.1
requests==2.0.0
simplejson==3.3.0
six==1.4.1
virtualenv==1.10.1
voluptuous==0.8.1
wsgiref==0.1.2

Answer 1

If you want to increase the upload size:

There are several settings in nginx, gunicorn and flask that govern this behavior to limit upload, what worked for me was:

flask

MEGABYTE = (2 ** 10) ** 2
app.config['MAX_CONTENT_LENGTH'] = None
# Max number of fields in a multi part form (I don't send more than one file)
# app.config['MAX_FORM_PARTS'] = ...
app.config['MAX_FORM_MEMORY_SIZE'] = 50 * MEGABYTE

Gunicorn:

--limit-request-line 0

Nginx: (Could be either server block, or route block (per route basis)

client_max_body_size 50M;

Answer 2

I recently encountered this with Flask 3.0.3, different cause. That flask uses Werkzeug, and a form variables limit in Werkzeug produced this error on the number of form variables, not on body size. The fix is:

from werkzeug import Request
Request.max_form_parts = 5000 # or whatever your max form size!

Answer 3

Note that if you are using the gunicorn server, you can set the following command line parameters to allow for larger requests to be made.

• --limit-request-line 0 will allow an unlimited size of the HTTP request size. The default is 4,094 bytes.
• --limit-request-fields 1000 will allow 1,000 header fields to be added to an HTTP request. The default is 100, and must be < 32,768.
• --limit-request-field_size 0 will allow an unlimited size of an HTTP request header. The default is 8,091 bytes.

Answer 4

Based on this github issue answers (https://github.com/benoitc/gunicorn/issues/1733#issuecomment-377000612)

@app.before_request
def handle_chunking():
    """
    Sets the "wsgi.input_terminated" environment flag, thus enabling
    Werkzeug to pass chunked requests as streams.  The gunicorn server
    should set this, but it's not yet been implemented.
    """

    transfer_encoding = request.headers.get("Transfer-Encoding", None)
    if transfer_encoding == u"chunked":
        request.environ["wsgi.input_terminated"] = True

Answer 5

Flask is closing the connection, you can set an error handler for the 413 error:

@app.errorhandler(413)
def request_entity_too_large(error):
    return 'File Too Large', 413

Now the client should get a 413 error, note that I didn't test this code.

Update:

I tried recreating the 413 error, and I didn't get a ConnectionError exception.

Here's a quick example:

from flask import Flask, request

app = Flask(__name__)

app.config['MAX_CONTENT_LENGTH'] = 1024


@app.route('/post', methods=('POST',))
def view_post():
    return request.data


app.run(debug=True)

After running the file, I used the terminal to test requests and sending large data:

>>> import requests
>>> r = requests.post('http://127.0.0.1:5000/post', data={'foo': 'a'})
>>> r
<Response [200]>
>>> r = requests.post('http://127.0.0.1:5000/post', data={'foo': 'a'*10000})
>>> r
<Response [413]>
>>> r.status_code
413
>>> r.content
'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>413 Request Entity Too Large</title
>\n<h1>Request Entity Too Large</h1>\n<p>The data value transmitted exceeds the capacity limit.</p>\n'

As you can see, we got a response from flask 413 error and requests didn't raise an exception.

By the way I'm using:

• Flask: 0.10.1
• Requests: 2.0.0

Answer 6

RFC 2616, the specification for HTTP 1.1, says:

10.4.14 413 Request Entity Too Large

The server is refusing to process a request because the request
entity is larger than the server is willing or able to process. The
server MAY close the connection to prevent the client from continuing the request.

If the condition is temporary, the server SHOULD include a Retry-
After header field to indicate that it is temporary and after what
time the client MAY try again.

This is what's happening here: flask is closing the connection to prevent the client from continuing the upload, which is giving you the Broken pipe error.