python manage.py syncdb fails with "ImproperlyConfigured: No module named 'psycopg2'

Question

I am currently going through these steps to install my first Digital Ocean server, with Ubuntu 14 x32 and Django 1.7, using Python 3.4. I'm new to both Ubuntu and Digitial Ocean, although I developed on Unix years ago.

I'm having a problem with Postgres and Django, when running syncdb.

---

(Note: I've installed my virtualenv with -p /usr/bin/python3.4, which is different than instructed. I don't know if this makes a difference.)

---

In step 8 it says to install psycopg2 with

pip install psycopg2

Which, as described in this question, has to be changed to

sudo /home/jeffy/django_files/django_test_venv/bin/pip install psycopg2

But it fails with

./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory

Full output:

(django_test_venv)jeffy@originaldjangster:~/django_files/django_test_venv/bin$ sudo /home/jeffy/django_files/django_test_venv/bin/pip install psycopg2
Downloading/unpacking psycopg2
  Downloading psycopg2-2.5.3.tar.gz (690kB): 690kB downloaded
  Running setup.py (path:/home/jeffy/django_files/django_test_venv/build/psycopg2/setup.py) egg_info for package psycopg2

Installing collected packages: psycopg2
  Running setup.py install for psycopg2
    Skipping implicit fixer: buffer
    Skipping implicit fixer: idioms
    Skipping implicit fixer: set_literal
    Skipping implicit fixer: ws_comma
    building 'psycopg2._psycopg' extension
    i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.5.3 (dt dec pq3 ext)" -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DPG_VERSION_HEX=0x090304 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/usr/include/python3.4m -I/home/jeffy/django_files/django_test_venv/include/python3.4m -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.3/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-3.4/psycopg/psycopgmodule.o -Wdeclaration-after-statement
    In file included from psycopg/psycopgmodule.c:27:0:
    ./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'i686-linux-gnu-gcc' failed with exit status 1
    Complete output from command /home/jeffy/django_files/django_test_venv/bin/python3.4 -c "import setuptools, tokenize;__file__='/home/jeffy/django_files/django_test_venv/build/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-jilc5e_m-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/jeffy/django_files/django_test_venv/include/site/python3.4:
    running install
running build
running build_py
creating build
creating build/lib.linux-i686-3.4
creating build/lib.linux-i686-3.4/psycopg2
copying lib/_range.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/extras.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/pool.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/extensions.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/errorcodes.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/psycopg1.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/__init__.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/_json.py -> build/lib.linux-i686-3.4/psycopg2
copying lib/tz.py -> build/lib.linux-i686-3.4/psycopg2
creating build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_dates.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_lobject.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_extras_dictcursor.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_transaction.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_green.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_psycopg2_dbapi20.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_bug_gc.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_bugX000.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_notify.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/dbapi20_tpc.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_types_basic.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/testconfig.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_with.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_quote.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_copy.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_module.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_types_extras.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/dbapi20.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_connection.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_async.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/__init__.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_cursor.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/testutils.py -> build/lib.linux-i686-3.4/psycopg2/tests
copying tests/test_cancel.py -> build/lib.linux-i686-3.4/psycopg2/tests
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
running build_ext
building 'psycopg2._psycopg' extension
creating build/temp.linux-i686-3.4
creating build/temp.linux-i686-3.4/psycopg
i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.5.3 (dt dec pq3 ext)" -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DPG_VERSION_HEX=0x090304 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/usr/include/python3.4m -I/home/jeffy/django_files/django_test_venv/include/python3.4m -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.3/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-3.4/psycopg/psycopgmodule.o -Wdeclaration-after-statement
In file included from psycopg/psycopgmodule.c:27:0:
./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory
 #include <Python.h>
                    ^
compilation terminated.
error: command 'i686-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /home/jeffy/django_files/django_test_venv/bin/python3.4 -c "import setuptools, tokenize;__file__='/home/jeffy/django_files/django_test_venv/build/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-jilc5e_m-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/jeffy/django_files/django_test_venv/include/site/python3.4 failed with error code 1 in /home/jeffy/django_files/django_test_venv/build/psycopg2
Storing debug log for failure in /home/jeffy/.pip/pip.log

I finally settled on this, which worked:

sudo apt-get install python-psycopg2

It then says to configure the Django DATABASE variable, in settings.py, to

"'ENGINE': 'django.db.backends.postgresql_psycopg2',"

and then run

python manange.py syncdb

syncdb is failing with

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2'

Full output:

(django_test_venv)jeffy@originaldjangster:~/django_files/django_test$ python manage.py syncdb
Traceback (most recent call last):
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 23, in <module>
    import psycopg2 as Database
ImportError: No module named 'psycopg2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 354, in execute
    django.setup()
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/__init__.py", line 21, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/apps/config.py", line 197, in import_models
    self.models_module = import_module(models_module_name)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2231, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1448, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/contrib/auth/models.py", line 40, in <module>
    class Permission(models.Model):
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/models/base.py", line 125, in __new__
    new_class.add_to_class('_meta', Options(meta, **kwargs))
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/models/base.py", line 300, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/models/options.py", line 166, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/__init__.py", line 40, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/utils.py", line 242, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/utils.py", line 108, in load_backend
    return import_module('%s.base' % backend_name)
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2231, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1448, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 27, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2'
(django_test_venv)jeffy@originaldjangster:~/django_files/django_test$

I confirmed that this file exists, which matches the DATABASE value:

/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2

(To repeat myself: I installed my virtualenv with -p /usr/bin/python3.4, which is different than instructed, although I don't know if this makes a difference.)

Answer 1

Old topic but could be useful to someone, just had the same issue : postgres with no psycopg2 using python 3.x

Had the same issue, after a bit of digging, this solved my problem :

https://pypi.org/project/psycopg2-binary/

pip install psycopg2-binary

Answer 2

Install python3x-dev and libpq-dev in your system before install psycopg2 in your virtualenv.

apt-get install python3.4-dev libpq-dev

then in your virtualenv

pip install psycopg2

Answer 3

As suggested in this answer, I attempted to import both psycopg2 and then postgresql_psycopg2 in IPython. Both returned an ImportError.

I saw the PYTHONPATH was empty, so I set it to the above directory

(django_test_venv)jeffy@originaldjangster:~/django_files/django_test$ echo $PYTHONPATH

(django_test_venv)jeffy@originaldjangster:~/django_files/django_test$ export PYTHONPATH=/home/jeffy/django_files/django_test_venv/lib/python3.4/site-packages

But again, both packages resulted in an ImportError in IPython. As in this answer (same question), I ran

sudo apt-get build-dep python-psycopg2

(Trying syncdb here failed the same way.)

and then (adjusted as required, which is mentioned in the question-post)

sudo /home/jeffy/django_files/django_test_venv/bin/pip install psycopg2

And now

python manage.py syncdb

works.