Reputation: 463
The setup is as follows
Nginx reverse proxy -> gunicorn -> django app
At a certain time, I observed that nginx was logging connection timed out, and sending back HTTP 502 error responses.
The reason was that the gunicorn child process did not respond for 30 seconds, and got restarted by gunicorn master.
I am trying to find out why my gunicorn process got stuck in sending back a response. From the logging, django app logic has completed successfully. I suspect that when django is returning the response to gunicorn via wsgi, something is going wrong.
Some other points:
1) The response size for these requests is large. ~ 200 kB. 2) When I replay these requests, they work fine. 3) This happenned on multiple servers for about 100 requests at the same time.
I am not able to pin point what went wrong. Any help would be appreciated!
Upvotes: 1
Views: 1971
Reputation: 4292
You could have gunicorn responding slow when you have some load(100 connections) as said. Try to increase the timeout for your proxy pass in Nginx by adding:
proxy_connect_timeout 300s;
proxy_read_timeout 300s;
If it will not help, try to change timout in gunicorn configuration:
NUM_WORKERS=3
TIMEOUT=120
exec gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--timeout $TIMEOUT \
--log-level=debug \
--bind=127.0.0.1:9000 \
--pid=$PIDFILE
Also there is --log-level=debug \
added to get more info from logs.
Upvotes: 1