How to use some field instead of primary keys in Django 1.6 model fixtures?

Question

I need to dump and load fixtures of model objects without usage of primary keys. The model is flat. I know about natural keys in Django, spent a lot of time reading the documentation, but all documentation has solutions only for usage natural keys instead of relations (fk / m2m). This is totally not what I need.

I need something like this:

(models.py)

class Template(models.Model):
    name = models.CharField(_('name'), max_length=100)
    content = models.TextField(_('content'), blank=True)

    def natural_key(self):
        return (self.name,)

(fixture1.json)

[
{
    "pk": null,
    "model": "dbtemplates.Template",
    "fields" {
        "content": "",
        "name": "product"
    }
}
]

And after command

./manage.py <SOME_LOADDATA_COMMAND> fixture1.json --natural

I need to update my Template object which has name "product" or insert it.

The standard Django commands don't do this. Please, help me with any solution. Maybe there are some libraries for this? I'm confusing.

Django 1.6. Python 2.7

Answer 1

Based on the answer of régis-b I wrote some code that allows use natural keys in Django 1.6 "loaddata" management command without upgrade to 1.7. I chose this way because a full upgrading of my project may be painful. This solution can be considered as a temporary.

tree structure:

├── project_main_app
│   ├── __init__.py
│   ├── backports
│   │   ├── __init__.py
│   │   └── django
│   │       ├── __init__.py
│   │       └── deserializer.py
│   └── monkey.py

project_main_app/backports/django/deserializer.py

from __future__ import unicode_literals

from django.conf import settings
from django.core.serializers import base
from django.core.serializers.python import _get_model
from django.db import models, DEFAULT_DB_ALIAS
from django.utils.encoding import smart_text
from django.utils import six


def Deserializer(object_list, **options):
    """
    Deserialize simple Python objects back into Django ORM instances.

    It's expected that you pass the Python objects themselves (instead of a
    stream or a string) to the constructor
    """
    db = options.pop('using', DEFAULT_DB_ALIAS)
    ignore = options.pop('ignorenonexistent', False)

    models.get_apps()
    for d in object_list:
        # Look up the model and starting build a dict of data for it.
        Model = _get_model(d["model"])
        data = {Model._meta.pk.attname: Model._meta.pk.to_python(d.get("pk", None))}
        m2m_data = {}
        model_fields = Model._meta.get_all_field_names()

        # Handle each field
        for (field_name, field_value) in six.iteritems(d["fields"]):

            if ignore and field_name not in model_fields:
                # skip fields no longer on model
                continue

            if isinstance(field_value, str):
                field_value = smart_text(field_value, options.get("encoding", settings.DEFAULT_CHARSET), strings_only=True)

            field = Model._meta.get_field(field_name)

            # Handle M2M relations
            if field.rel and isinstance(field.rel, models.ManyToManyRel):
                if hasattr(field.rel.to._default_manager, 'get_by_natural_key'):
                    def m2m_convert(value):
                        if hasattr(value, '__iter__') and not isinstance(value, six.text_type):
                            return field.rel.to._default_manager.db_manager(db).get_by_natural_key(*value).pk
                        else:
                            return smart_text(field.rel.to._meta.pk.to_python(value))
                else:
                    m2m_convert = lambda v: smart_text(field.rel.to._meta.pk.to_python(v))
                m2m_data[field.name] = [m2m_convert(pk) for pk in field_value]

            # Handle FK fields
            elif field.rel and isinstance(field.rel, models.ManyToOneRel):
                if field_value is not None:
                    if hasattr(field.rel.to._default_manager, 'get_by_natural_key'):
                        if hasattr(field_value, '__iter__') and not isinstance(field_value, six.text_type):
                            obj = field.rel.to._default_manager.db_manager(db).get_by_natural_key(*field_value)
                            value = getattr(obj, field.rel.field_name)
                            # If this is a natural foreign key to an object that
                            # has a FK/O2O as the foreign key, use the FK value
                            if field.rel.to._meta.pk.rel:
                                value = value.pk
                        else:
                            value = field.rel.to._meta.get_field(field.rel.field_name).to_python(field_value)
                        data[field.attname] = value
                    else:
                        data[field.attname] = field.rel.to._meta.get_field(field.rel.field_name).to_python(field_value)
                else:
                    data[field.attname] = None

            # Handle all other fields
            else:
                data[field.name] = field.to_python(field_value)

        # The key block taken from Django 1.7 sources
        obj = build_instance(Model, data, db)
        yield base.DeserializedObject(obj, m2m_data)

# This is also taken from Django 1.7 sources
def build_instance(Model, data, db):
    """
    Build a model instance.
    If the model instance doesn't have a primary key and the model supports
    natural keys, try to retrieve it from the database.
    """
    obj = Model(**data)
    if (obj.pk is None and hasattr(Model, 'natural_key') and
            hasattr(Model._default_manager, 'get_by_natural_key')):
        natural_key = obj.natural_key()
        try:
            obj.pk = Model._default_manager.db_manager(db).get_by_natural_key(*natural_key).pk
        except Model.DoesNotExist:
            pass

    return obj

project_main_app/monkey.py

def patch_all():
    import django.core.serializers.python
    import project_main_app.backports.django.deserializer
    # Patch the Deserializer
    django.core.serializers.python.Deserializer = project_main_app.backports.django.deserializer.Deserializer

project_main_app/init.py

from project_main_app.monkey import patch_all
patch_all()

So after this I just add some things and my model becomes like

class TemplateManager(models.Manager):
    """1"""
    def get_by_natural_key(self, name):
        return self.get(name=name)


class Template(models.Model):
    name = models.CharField(_('name'), max_length=100)
    content = models.TextField(_('content'), blank=True)
    objects = TemplateManager()  # 2        

    def natural_key(self):
        """3"""
        return (self.name,)

and if fixtures has an empty pk like

[
{
    "pk": null,
    "model": "dbtemplates.Template",
    "fields": {
        "content": "Some content",
        "name": "product"
    }
}
]

the standard command ./manage.py loaddata dbtemplates.Template updates or inserts object matching by name field.

Warning: all of natural key components (like a "name" in my case) must have unique values in the database. The proper way is to set them unique by adding the argument "unique=True" when defining a model.

Answer 2

Django 1.6 does not provide a way to dump data with natural primary keys, but Django 1.7 does.

• Django 1.6 docs: https://docs.djangoproject.com/en/1.7/ref/django-admin/#dumpdata-app-label-app-label-app-label-model
• Django 1.7 docs (note the --natural-primary option): https://docs.djangoproject.com/en/1.7/ref/django-admin/#dumpdata-app-label-app-label-app-label-model

Unfortunately, the use_natural_primary_keys keyword argument is not supported by the base Django 1.6 serializer: https://github.com/django/django/blob/1.6.11/django/core/serializers/base.py#L20

So I suggest you either upgrade to django 1.7 (which I totally understand is not always possible) or you write your own serializer, drawing inspiration from the base Django 1.7 serializer (https://github.com/django/django/blob/1.7.11/django/core/serializers/base.py).