Circular dependency in Django Rest Framework serializers

Question

I'm fighting with circular dependencies within serializers in my web API written using Django Rest Framework 3. Whereas I know that circular dependencies in a project is almost always a sign of bad design, I can't find a decent way of avoiding it without making the app a big monolithic nightmare.

A simple stripped down example pictures well enough what happens in all places I'm having the similar problem.

Let's have two simple models in two apps:

Profiles app

# profiles/models.py

from images.models import Image

class Profile(models.Model):
    name = models.CharField(max_length=140)  

    def recent_images(self):
        return Image.objects.recent_images_for_user(self)

Images app

# images/models.py

class Image(models.Model):
    profile = models.ForeignKey('profiles.Profile')
    title = models.CharField(max_length=140)

Following the principle of fat models I often use multiple imports in my models to allow easy retrieval of related objects using methods on Profile, but that rarely causes circular dependencies, since I rarely do the same from the other end.

The problem begins when I try to add serializers to the bunch. To make the API footprint small and limit the amount of necessary calls to the minimum, I want to serialize on both ends some of the related objects in their simplified forms.

I want to be able to retrieve profiles on /profile endpoint that will have simplified info about few recent images created by the user nested. Also, when retrieving images from /images endpoint I'd like to have profile info embedded in the image JSON.

To achieve this and avoid recursive nesting, I have two serializers - one that nests related objects, and one that does not, for both apps.

Profiles app

# profiles/serializers.py

from images.serializers import SimplifiedImageSerializer

class SimplifiedProfileSerializer(serializers.Serializer):
    name = serializers.CharField()

class ProfileSerializer(SimplifiedProfileSerializer):
    recent_images = SimplifiedImageSerializer(many=True)

Images app

# images/serializers.py

from profiles.serializers import SimplifiedProfileSerializer

class SimplifiedImageSerializer(serializers.Serializer):
    title = serializers.CharField()

class ImageSerializer(SimplifiedImageSerializer):
    profile = SimplifiedProfileSerializer()

The expected behaviour is to get the following JSON results:

Profiles app at /profiles

[{
    'name': 'Test profile',
    'recent_images': [{
        'title': 'Test image 1'
    }, {
        'title': 'Test image 2'
    }]
]]

Images app at /images

[{
    'title': 'Test image 1',
    'profile': {
        'name': 'Test profile'
    }
},
{
    'title': 'Test image 2',
    'profile': {
        'name': 'Test profile'
    }
}]

but then I hit the wall with circular imports of the serializers.

I feel that joining those two apps into one is definitely not the road to take - after all, images are something completely different from user profiles.

The serializers also in my view should belong to their respective apps.

The only way to go around this problem I found as of now is import in the method as follows:

class ImageSerializer(SimplifiedProfileSerializer):
    profile = SerializerMethodField()

    def get_profile(self, instance):
        from profiles.serializers import SimplifiedProfileSerializer
        return SimplifiedProfileSerializer(instance.profile).data

but that feels like an ugly, ugly, uuuugly hack.

Could you please share your experience with similar problems?

Thanks!

Answer 1

I just created this utility class to handle circular import problem:

from django.utils.module_loading import import_string
from django.utils.functional import cached_property
from rest_framework.serializers import RelatedField


class DynamicRelatedField(RelatedField):
    def __init__(self, serializer_path=None, **kwargs):
        assert serializer_path is not None, 'The `serializer_path` argument is required.'
        assert kwargs['read_only'], 'Only readonly fields are supported for DynamicRelatedField'
        self.serializer_path = serializer_path
        super().__init__(**kwargs)

    @cached_property
    def serializer_object(self):
        serializer_class = import_string(self.serializer_path)
        return serializer_class()

    def to_representation(self, obj):
        return self.serializer_object.to_representation(obj)

    def to_internal_value(self, data):
        return None

And then, I can use it like the following:

class ProfessorDetailsSerializer(serializers.ModelSerializer):
    courses = DynamicRelatedField('courses.serializers.CourseDetailsSerializer', many=True, read_only=True)

    class Meta:
        model = Professor
        fields = ('id', 'name', 'courses')

Notes

• I used cached_property to prevent serializer_object creation on each instance when using many=True.
• This snippet doesn't support writable fields, but adding this feature won't take much work ;-)

Answer 2

I'd take a different approach as you do have coupling one way or another. I'd go with defining the serializer I actually use within the application itself.

Profile application:

# profiles/serializers.py

class SimplifiedImageSerializer(serializers.Serializer):
    title = serializers.CharField()

class ProfileSerializer(SimplifiedProfileSerializer):
    recent_images = SimplifiedImageSerializer(many=True)

Image application:

# images/serializers.py

class SimplifiedProfileSerializer(serializers.Serializer):
    name = serializers.CharField()

class ImageSerializer(SimplifiedImageSerializer):
    profile = SimplifiedProfileSerializer()

Answer 3

You should consider having a look at Specifying nested serialization in the Rest Framework documentation. The usage of depth meta attribute enables you to retrieve related objects to the depth you set.

It is very convenient to avoid using serializers in both sides and thus having ImportError caused by cycles.

The default ModelSerializer uses primary keys for relationships, but you can also easily generate nested representations using the depth option:

class AccountSerializer(serializers.ModelSerializer):
    class Meta:
        model = Account
        fields = ['id', 'account_name', 'users', 'created']
        depth = 1

Answer 4

I have suffered a lot with Django Serializers Circular Dependency issue and found only two ways to solve it.

1. Arranging my code in a way so that I don't have to face circular dependency (which was not possible in my case)
2. Creating separate serializer class with the serializer I need and using this new serializer where I needed. This one may not be the most efficient one but this solved my issue.

Answer 5

you can do a local import of serializers like this:

class MySerializer(Serializer):
    from app.core.serializers import AnotherSerializer

Do that in both of your imports. No need to use sys.modules

That is, as mentioned by Sebastian Wozny, that you don't have a logical circular dependancy

Answer 6

Separating usual and nested serializers does the trick for me.

For your structure it will be something like:

Profiles app

# profiles/serializers/common.py

from images.serializers.nested import SimplifiedImageSerializer

class ProfileSerializer(SimplifiedProfileSerializer):
    recent_images = SimplifiedImageSerializer(many=True)

And nested:

# profiles/serializers/nested.py

class SimplifiedProfileSerializer(serializers.Serializer):
    name = serializers.CharField()

Images app

# images/serializers/common.py

from profiles.serializers.nested import SimplifiedProfileSerializer

class ImageSerializer(SimplifiedImageSerializer):
    profile = SimplifiedProfileSerializer()

And nested:

# images/serializers/nested.py

class SimplifiedImageSerializer(serializers.Serializer):
    title = serializers.CharField()

Answer 7

In my opinion your code is fine, because you do not have a logic circular dependency.

Your ImportError is only raised because of the way import() evaluates top level statements of the entire file when called.

However, nothing is impossible in python...

There is a way around it if you positively want your imports on top:

From David Beazleys excellent talk Modules and Packages: Live and Let Die! - PyCon 2015, 1:54:00, here is a way to deal with circular imports in python:

try:
    from images.serializers import SimplifiedImageSerializer
except ImportError:
    import sys
    SimplifiedImageSerializer = sys.modules[__package__ + '.SimplifiedImageSerializer']

This tries to import SimplifiedImageSerializer and if ImportError is raised, because it already is imported, it will pull it from the importcache.

PS: You have to read this entire post in David Beazley's voice.