Django, Python 3 & Postgres - mod_wsgi issue

Question

I'm trying to configure a Django test application for a personal project. I have previous Django experience, but that was all with Python 2.7.x. I'd like to start using 3.x when possible, and this project seems like a good way to start.

Backend is a standard Postgres 9.4 installation, Apache 2.4, Python 3.4 in a virtualenvwrapper, operating system is OSX 10.10. mod_wsgi and mod_wsgi3 have both been installed via homebrew.

However, there are issues getting mod_wsgi and mod_wsgi3 to work with python 3. Specifically, ./manage.py runserver 8080 works fine, but when I try to use a virtualhost configuration with Apache, I get

mod_wsgi (pid=29906): Target WSGI script '/Users/jason/projects/geocode_django/geodjango/geodjango/wsgi.py' cannot be loaded as Python module.
mod_wsgi (pid=29906): Exception occurred processing WSGI script '/Users/jason/projects/geocode_django/geodjango/geodjango/wsgi.py'.
Traceback (most recent call last):
    File "/Users/jason/projects/geocode_django/geodjango/geodjango/wsgi.py", line 28, in <module>
    application = get_wsgi_application()
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/core/wsgi.py", line 14, in       get_wsgi_application
      django.setup()
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/__init__.py", line 21, in setup
      apps.populate(settings.INSTALLED_APPS)
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
     app_config.import_models(all_models)
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/apps/config.py", line 197, in import_models
      self.models_module = import_module(models_module_name)
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
       __import__(name)
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/contrib/auth/models.py", line 40, in <module>
       class Permission(models.Model):
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/models/base.py", line 125, in __new__
       new_class.add_to_class('_meta', Options(meta, **kwargs))
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/models/base.py", line 300, in add_to_class
       value.contribute_to_class(cls, name)
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/models/options.py", line 166, in contribute_to_class
       self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/__init__.py", line 40, in __getattr__
       return getattr(connections[DEFAULT_DB_ALIAS], item)
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/utils.py", line 242, in __getitem__
        backend = load_backend(db['ENGINE'])
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/utils.py", line 108, in load_backend
        return import_module('%s.base' % backend_name)
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
        __import__(name)
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 2, in <module>
        from django.db.backends.postgresql_psycopg2.base import DatabaseWrapper as Psycopg2DatabaseWrapper
    File "/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 27, in <module>
        raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
          ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/  site-packages/psycopg2/_psycopg.so, 2): Symbol not found: _PyBytes_Type
        Referenced from: /Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/psycopg2/_psycopg.so
          Expected in: flat namespace
          in /Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages/psycopg2/_psycopg.so

My wsgi.py file is

import os, sys, site

site.addsitedir("/Users/jason/.virtualenvs/geodev_env3/lib/python3.4/site-packages")

sys.path.append("/Users/jason/projects/geocode_django/geodjango")
sys.path.append("/Users/jason/projects/geocode_django/geodjango/geodjango")

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "geodjango.settings")

with open("/Users/jason/.virtualenvs/geodev_env3/bin/activate_this.py") as f:
    code = compile(f.read(), "/Users/jason/.virtualenvs/geodev_env3/bin/activate_this.py", "exec")
    exec(code, dict(__file__="/Users/jason/.virtualenvs/geodev_env3/bin/activate_this.py"))

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

and virtualhost file is:

LoadModule wsgi_module /usr/local/Cellar/mod_wsgi3/3.5/libexec/mod_wsgi.so

<VirtualHost *:80>
    ServerName      dev.geocode.com
    ServerAlias     geocode.com
    ServerAdmin     jason@someschool.edu

    DocumentRoot    "/Users/jason/projects/geocode_django"
    Alias /static/ /Users/jason/projects/geocode_django/static/

    WSGIScriptAlias / /Users/jason/projects/geocode_django/geodjango/geodjango/wsgi.py

    <Directory /Users/jason/projects/geocode_django/static >
            require all granted
    </Directory>

    <Directory /Users/jason/projects/geocode_django/geodjango/geodjango >
            <Files wsgi.py >
                    Require all granted
            </Files>

    </Directory>

    CustomLog /Users/jason/projects/geocode_django/logs/access.log combined
    ErrorLog /Users/jason/projects/geocode_django/logs/error.log

</VirtualHost>

My issue is that if I have a python 2.7 virtualenv with the same virtualhost with the LoadModule pointing to

LoadModule wsgi_module /usr/local/Cellar/mod_wsgi/4.4.7/libexec/mod_wsgi.so

and replace

with open("/Users/jason/.virtualenvs/geodev_env3/bin/activate_this.py") as f:
    code = compile(f.read(), "/Users/jason/.virtualenvs/geodev_env3/bin/activate_this.py", "exec")
    exec(code, dict(__file__="/Users/jason/.virtualenvs/geodev_env3/bin/activate_this.py"))

in wsgi.py with

exec(open("/Users/jasonjohns/.virtualenvs/geocode_env/bin/activate_this.py").read())

I can load up the site with the URL dev.geocode.com. Otherwise, I get an error 500 page with the error output in the log file.

I originally thought this was an issue with psycopg2, and filed a bug report. However, the dev closed the issue as being concerned with mod_wsgi, not psycopg.

Short of compiling mod_wsgi for my local environment, is there a way to fix this issue?

Answer 1

With a hint from Graham Dumpleton via the mod_wsgi google group, a solution was found. The core issue was that the version of mod_wsgi installed by both brew install mod_wsgi and pip install mod_wsgi were all compiled for Python 2.7.x.

Another thing. pip, on a system where python --version returns Python 2.7.6, will use that Python version for building any install commands. If you have Python 3.x, alongside 2.7.x and python --version returns Python 3.x.x, pip3 install ... will download 3.x compatible code and build for that Python version.

The solution was to remove both installations via

brew uninstall mod_wsgi
pip uninstall mod_wsgi

and install via

pip3 install mod_wsgi

As a result, the LoadModule location for the Apache configuration file was changed to

/usr/local/lib/python3.4/site-packages/mod_wsgi-4.4.9-py3.4-macosx-10.10-x86_64.egg/mod_wsgi/server/mod_wsgi-py34.so

Note the python3.4 location in the path.

After making these changes and restarting Apache, the site loads as expected.