django app under uwsgi failing to start when using postgresql database

Question

I setup an app in Ubuntu 14.04 on AWS -> NGINX -> UWSGI -> DJANGO APP. The app runs completely fine when using sqllite3, but when I use django.db.backends.postgresql_psycopg2w as my database engine it fails to load and I find this in uwsgi logs:

- *** Starting uWSGI 2.0.12 (64bit) on [Mon Jan 18 05:36:25 2016] ***
 - compiled with version: 4.8.4 on 18 January 2016 04:10:30
 - os: Linux-3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015
 - nodename: ip-172-31-55-58
 - machine: x86_64
 - clock source: unix
 - detected number of CPU cores: 1
 - current working directory: /var/www/carzumer_api
 - detected binary path: /usr/local/bin/uwsgi
 - !!! no internal routing support, rebuild with pcre support !!!
 - setgid() to 33
 - setuid() to 33
 - chdir() to /var/www/carzumer_api/src/
 - your processes number limit is 7862
 - limiting address space of processes...
 - your process address space limit is 134217728 bytes (128 MB)
 - your memory page size is 4096 bytes
 -  *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
 - detected max file descriptor number: 1024
 - lock engine: pthread robust mutexes
 - thunder lock: disabled (you can enable it with --thunder-lock)
 - uwsgi socket 0 bound to UNIX address /var/www/carzumer_api/run/uwsgi.sock fd 6
 - Python version: 2.7.6 (default, Jun 22 2015, 18:01:27)  [GCC 4.8.2]
 - Set PythonHome to /var/www/carzumer_api/env
 - *** Python threads support is disabled. You can enable it with --enable-threads ***
 - Python main interpreter initialized at 0x2047200
 - your server socket listen backlog is limited to 100 connections
 - your mercy for graceful operations on workers is 60 seconds
 - mapped 363840 bytes (355 KB) for 4 cores
 - *** Operational MODE: preforking ***
Traceback (most recent call last):
  File "./carzumer_api/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
    django.setup()
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/contrib/auth/models.py", line 41, in <module>
    class Permission(models.Model):
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 139, in __new__
    new_class.add_to_class('_meta', Options(meta, **kwargs))
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 324, in add_to_class
    value.contribute_to_class(cls, name)
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 250, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/__init__.py", line 36, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/utils.py", line 240, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/www/carzumer_api/env/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 24, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: libgpg-error.so.0: failed to map segment from shared object: Cannot allocate memory
 - unable to load app 0 (mountpoint='') (callable not found or import error)
 - *** no app loaded. going in full dynamic mode ***
 - *** uWSGI is running in multiple interpreter mode ***
 - spawned uWSGI master process (pid: 11351)
 - spawned uWSGI worker 1 (pid: 11354, cores: 1)
 - spawned uWSGI worker 2 (pid: 11355, cores: 1)
 - --- no python application found, check your startup logs for errors ---
{address space usage: 81760256 bytes/77MB} {rss usage: 21651456 bytes/20MB} [pid: 11355|app: -1|req: -1/1] 127.0.0.1 () {32 vars in 351 bytes} [Mon Jan 18 05:37:50 2016] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 82 bytes (0 switches on core 0)

I have installed the following packages using apt-get: libpq-dev python-dev postgresql postgresql-contrib nginx python-pip

Also the psycopg2==2.6.1 package is installed in my venv.

Can anyone point me in the right direction, I have searched for hours now and have no idea why it's failing.

Answer 1

I finally figured out what my problem was. In my uwsgi.ini file I had

limit-as = 128

This switch limits the address space of uwsgi process to 128mb. I guess that was not enough for PostgreSQL driver. After removing the switch everything works like a charm. Although it's better to find a value that works instead of just removing the value altogether.

Answer 2

Root exception is this:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: libgpg-error.so.0: failed to map segment from shared object: Cannot allocate memory

From what I found, it may be a 32/64-bit issue. For example, see this page:

https://serverfault.com/questions/329418/cannot-allocate-memory-error-for-psycopg2-and-libgcrypt-so-on-debian-6-64bit

You might want to use the file tool and check the architectures of your libs (specifically _psycopg.so and libgpg-error.so.0). Also compare with python -c "import platform; print platform.architecture()"