CODEWITHSUNDEEP
djangoamazon-web-servicesnginxamazon-ec2gunicorn
h4x0rjax
h4x0rjax

Reputation: 359

http 403 error + "readv() failed (104: Connection reset by peer) while reading upstream"

Preface: I'm running nginx + gunicorn + django on an amazon ec2 instance using s3boto as a default storage backend. I am free tier. The ec2 security group allows: http, ssh, & https.

I'm attempting to send a multipart/form-data request containing a single element: a photo. When attempting to upload the photo, the iPhone (where the request is coming from) hangs. The photo is around 9.5 MB in size.

When I check the nginx-access.logs:

"POST /myUrl/ HTTP/1.1" 400 5 "-""....

When I check the nginx-error.logs:

[error] 5562#0: *1 readv() failed (104: Connection reset by peer) while reading upstream, client: my.ip.addr.iphone, server: default, request: "POST /myUrl/ HTTP/1.1", upstream: "http://127.0.0.1:8000/myUrl/", host: "ec2-my-server-ip-addr.the-location-2.compute.amazonaws.com"

[info] 5562#0: *1 client my.ip.addr.iphone closed keepalive connection

I really cannot figure out why this is happening... I have tried changing the /etc/nginx/sites-available/default timeout settings...

server { ...

    client_max_body_size 20M;
    client_body_buffer_size 20M;


    location / {
       keepalive_timeout 300;
       proxy_read_timeout 300;
    }
}

Any thoughts?

Upvotes: 2

Views: 3099

Answers (1)

Colin Nichols
Colin Nichols

Reputation: 714

EDIT: After talking on IRC a little more, his problem is the 403 itself, not the nginx error. Leaving my comments on the nginx error below, in case anyone else stumbles into it someday.

I ran into this very problem last week and spent quite a while trying to figure out what was going on. See here: https://github.com/benoitc/gunicorn/issues/872

Basically, as soon as django sees the headers, it knows that the request isn't authenticated. It doesn't wait for the large request body to finish uploading; it responds immediately, and gunicorn closes the connection right after. nginx keeps sending data, and the end result is that gunicorn sends a RST packet to nginx. Once this happens, nginx cannot recover and instead of sending the actual response from gunicorn/django, it sends a 502 Bad Gateway.

I ended up putting in a piece of middleware that acecsses a couple fields in the django request, which ensures that the entire request body is downloaded before Django sends a response:

checker = re.compile(feed_url_regexp)

class AccessPostBodyMiddleware:

    def process_request(self, request):
        if checker.match(request.path.lstrip('/')) is not None:
            # just need to access the request info here
            # not sure which one of these actually does the trick.
            # This will download the entire request,
            # fixing this random issue between gunicorn and nginx
            _ = request.POST
            _ = request.REQUEST
            _ = request.body
        return None

However, I do not have control of the client. Since you do (in the form of your iphone app), maybe you can find a way to handle the 502 Bad Gateway. That will keep your app from having to send the entire request twice.

Upvotes: 2

Related Questions