CODEWITHSUNDEEP
pythondjangonginxdockergunicorn
Joe Mornin
Joe Mornin

Reputation: 9134

Debugging silent failure in Gunicorn/NGINX/Django

I'm running a Django project with Gunicorn. I'm using Nginx as a reverse proxy. For the most part, everything works fine, but there's one Django view that causes Gunicorn to fail silently.

Details of the problem are below, but first, here's the Django view that's causing the problem:

def jobs_all(request):
    if not request.user.is_superuser: raise Http404
    jobs = Job.objects.all().order_by('-date_created')
    return render(request, 'monitor/jobs_all.html', {
        'jobs': jobs,
    })

If I change 'jobs': jobs, to 'jobs': [],, it works. So I gather that the problem involves passing those QuerySet results to the template.

The error that displays in the browser is Nginx's 502 Bad Gateway error. My Nginx error log reads:

2017/01/22 05:17:25 [error] 22#0: *26 upstream prematurely closed connection while reading response header from upstream, client: 12.34.56.78, server: www.example.com, request: "GET /monitor/jobs/all/ HTTP/1.1", upstream: "http://127.0.0.1:8000/monitor/jobs/all/", host: "www.example.com", referrer: "https://www.example.com/monitor/"

OK, so it looks like Gunicorn is timing out or closing somehow. But I don't see how. The error occurs within a few seconds, which shouldn't be a problem under my Nginx and Gunicorn configuration. I don't see any other error messages.

Here is my command to start Gunicorn:

$ /usr/local/bin/gunicorn -b 127.0.0.1:8000 --keep-alive 43200 -w 4 --log-level=DEBUG mydjangoproject.wsgi --timeout=43200

(In production, I start Gunicorn by using Supervisor to invoke a similar command but with a different log level and path to a log file. I've checked the Gunicorn logs; they're silent about this error.)

My Nginx config is:

server {
    listen 443 ssl;
    server_name www.example.com;
    access_log /dev/null;
    error_log /logs/nginx/nginx.error.log;

    ssl_certificate /code/ssl/ssl-bundle.crt;
    ssl_certificate_key /code/ssl/server.key;
    ssl_dhparam /code/ssl/dhparams.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    server_tokens off;

    if ($request_method !~ ^(GET|HEAD|POST)$ ) {
        return 405;
    }

    location /static/ {
        root /;
    }

    # TODO add media directory

    client_max_body_size 100M;

    location / {

        client_body_buffer_size 500K;
        client_max_body_size 100M;
        keepalive_timeout 43200;

        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 43200;
        proxy_read_timeout 43200;
        proxy_pass http://127.0.0.1:8000/;
    }
}

I've also switched on DEBUG=True in my Django project. No luck there—I still get the Nginx 502 Bad Gateway error (whereas with other errors, like 404s, I see the Django error page as expected).

When I trigger this error while running Gunicorn with --log-level=DEBUG, the output looks like this:

[2017-01-22 05:17:22 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:23 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:24 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:25 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:25 +0000] [1100] [INFO] Booting worker with pid: 1100
[2017-01-22 05:17:26 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:26 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:27 +0000] [1042] [DEBUG] 4 workers
[2017-01-22 05:17:28 +0000] [1042] [DEBUG] 4 workers

It appears that the worker handling the request silently dies and then silently respawns.

If I run the Django dev server (with python manage.py runserver 127.0.0.1:8000) instead of Gunicorn, the view works fine. So this appears to be a problem with my Gunicorn configuration rather than my Python/Django code.

I don't think it matters here, but just in case: everything is running inside a Docker container.

Upvotes: 1

Views: 2000

Answers (1)

Joe Mornin
Joe Mornin

Reputation: 9134

The problem was lack of system resources. I was running the code on a Digital Ocean droplet with 512MB of RAM. I ran htop while making a request to the Django view mentioned in the question, and I saw that RAM usage maxed out right when the Gunicorn worker died. Once I upgraded to a droplet with 1GB of RAM, this problem disappeared. Interesting that the Django dev server handled the request where Gunicorn choked.

Upvotes: 2

Related Questions