How to run Django units test with default and unmanaged database?

Question

I have a Django project with a default database used for storing things like user, orders etc.

We also have an unmanaged database. Now when you run Django test they try to make test databases, but since we have an unmanaged db we cannot do this. I cannot create migrations of this db since that will land 300 errors about clashing reverse accessor.

We use Docker and automatically spin up this unmanaged database and fill it with some mock data. This one is used for development and such. I would like the unit test to use this one for testing.

I tried things like creating migrations but since the reverse accessor issue this is not possible.

Is there a way to use the unmanaged database for unit testing? The test_default database which Django creates is fine, but I cannot create a test_unmanaged database.

Answer 1

We use a setup with managed and unmanaged tables in the same database, which might also work for your use case:

We have a script to generate the test database from two dumps: test_structure.sql an test_fixtures.sql. The former contains the structure of the database at a certain point in time, including all unmanaged tables. The latter contains any data you might need in the unmanaged tables during testing, and the contents of the django_migrations table. We dump test_fixtures.sql using a generated list of COPY (SELECT * FROM {table}) TO STDOUT; statements, for example: COPY (SELECT * FROM obs_00.django_migrations) TO STDOUT WITH NULL '--null--';.

The output from psql -c {copy_statement} is transformed to INSERT statements using a function like this:

def csv2sqlinsert(table_name, data):
    """
    Convert TSV output of  COPY (SELECT * FROM {table}) TO STDOUT
    to                     INSERT INTO {table} VALUES (), ()...();
    """

    def is_int(val):
        try:
            return "{}".format(int(val)) == val
        except ValueError:
            return False

    def column(data):
        if data == "--null--":
            return "null"
        elif is_int(data):
            return data
        else:
            return "'{}'".format(data.replace("'", "''"))  # escape quotes

    rows = [row.split("\t") for row in data.decode().split("\n") if len(row) > 1]

    if len(rows) == 0:
        return f"-- no data for {table_name}\n"

    data = ",\n".join("({})".format(",".join(column(col) for col in row)) for row in rows)

    ret = ""
    ret += f"-- {table_name} ({len(rows)} rows)\n"
    ret += f"INSERT INTO {table_name} VALUES\n{data};\n"

    return ret

In reality this function is more complicated, also simplifying our postgis geometries and truncating large text fields to save space.

creating the test db

Define the test db name in settings_test.py:

DATABASES["default"].update({
    "NAME": "django_test_db",
    "TEST": {"NAME": "django_test_db",},
})

With the two files above, (re)creating the test database looks like this:

dropdb django_test_db
createdb django_test_db
psql -d django_test_db -f test_structure.sql
psql -d django_test_db < test_fixtures.sql

We now have the state of the database at the moment of the dump. Because there might be new migrations, we let django migrate:

./manage.py migrate --settings=settings_test

Running the tests

Now we can run the tests using ./manage.py test --settings=settings_test. Because recreating the database every test run might take a considerable amount of time, adding --keepdb will save you a lot of time waiting for the test database restore procedure.

We've amended manage.py like this so we cannot forget:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    if len(sys.argv) > 1 and sys.argv[1] == "test":
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings_test")
        cmd = sys.argv + ["--keepdb"]
    else:
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
        cmd = sys.argv

    from django.core.management import execute_from_command_line
    execute_from_command_line(cmd)