CODEWITHSUNDEEP
pythondjangomongodbnginxuwsgi
dotslash
dotslash

Reputation: 2061

Django, nginx and uWSGI : uWSGI workers keep dying and reloading

We are using mongodb in our application. Our python process (django app) reads data from mongodb and based on some calculations it sends an HttpResponse. Now the problem is, the data read from mongodb is quite large. Every query size is nearly 900MB (1 Gb in some cases). Our python process goes out of memory and throws an MemoryError, due to which uWSGI worker process dies and spawns again.

My uwsgi.conf :

        [uwsgi]
chdir=/var/www/MERU/Meru/deviceapp/meru_device/
pythonpath=/var/www/MERU/Meru/deviceapp/meru_device/
env=DJANGO_SETTINGS_MODULE=meru_device.settings
module=meru_device.wsgi:application
uid=www-data
gid=www-data
socket=/tmp/da.sock
daemonize=/tmp/da.log
processes=2
master=true
listen=512
harakiri=80
memory-report=true
limit-as=1024
reload-on-as=1024
reload-on-rss=512
pidfile=/tmp/da.pid
chmod-socket=666
vacuum=true
max-requests=5000

My uwsgi_params file :

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;

My nginx.conf :

upstream django {
    server unix:///tmp/da.sock;
}

server {
    listen      8000;
    server_name <my-elastic-ip>;
    charset     utf-8;

    # max upload size
    client_max_body_size 75M;

    location /meru-nms {
        uwsgi_pass  django;
        include     /var/www/MERU/Meru/deviceapp/meru_device/uwsgi_params; # the uwsgi_params file you installed
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, OPTIONS';
    add_header 'Access-Control-Allow-Headers' '*';
    add_header 'Access-Control-Max-Age' 1728000;
    }
}

Do we need to increase memory limit for the python process ? Or do we need to increase address space limit for uwsgi ? Can someone point us in the right direction ?

Upvotes: 1

Views: 2006

Answers (1)

Gill Bates
Gill Bates

Reputation: 15137

I would recommend you to setup memory profiling for your application - you can write simple middleware that runs your views with profiler. As profiler you can use memory_profiler.

Upvotes: 1

Related Questions