CODEWITHSUNDEEP
djangodjango-rest-framework
ARR
ARR

Reputation: 2308

Django rest framework extremly slow (recursive relations)

For my project I started to use Laravel for the api than I switched to Django / django rest framework, I did this to gain more speed as I need to query large data.

Now I got the following situation: I got a "Group" which has "Subjects" and which has a recursive relation.

Now a group can have like 2000+ subjects(including the descendent subjects) a parent subject has +/- 30 subjects.

This is my code:

serializers

class RecursiveField(serializers.Serializer):
    def to_representation(self, value):
        serializer = self.parent.parent.__class__(value, context=self.context)
        return serializer.data

class SubjectSerializer(serializers.ModelSerializer):
    parent_of = RecursiveField(many=True, read_only=True)

    class Meta:
        model = Subject
        fields = ("id", "name", "parent_of", "parent")

class GroupSerializer(serializers.ModelSerializer):
    subjects = SubjectSerializer(many=True, read_only=True)

    class Meta:
        model = Group
        fields = ("id", "name", "subjects")

    def setup_eager_loading(cls, queryset):
        return queryset.prefetch_related("subjects")

views

class GroupViewSet(ModelViewSet):

    class Paginator(BasePaginator):
        model = Group

    queryset = Group.objects.all()
    serializer_class = serializers.GroupSerializer
    pagination_class = Paginator

    def get_queryset(self):
        return self.get_serializer().setup_eager_loading(GroupViewSet.queryset)

I tested the same request with the laravel api and its much faster, still noticeable slow but its ok(5-10 secs). With django rest framework its too slow (1 minute +/-), and thats just a page with 1 group that has 2500 subjects.

I do know what takes long, the RecursiveField class, because when I remove that the query is done in less than 2 seconds. So my question is whats the main cause, because it's creates a recursive relation (I doubt)? Or is it because I don't prefetch?

And ofcourse whats the best way to do this?

Thank you

Upvotes: 1

Views: 904

Answers (2)

Fabio Caccamo
Fabio Caccamo

Reputation: 1971

The problem is not DRF, but the data structure itself.

It is very slow in django to query all ancestors/descendants recursively, your should use a more efficient data structure.

For the same reason I wrote django-treenode, it performs tree operations without query the db. You can read the docs here: https://github.com/fabiocaccamo/django-treenode

Upvotes: 0

schillingt
schillingt

Reputation: 13731

You have a few options, but I don't think any are great. Recursive queries aren't very well supported with Django.

  1. Rework your data model to prevent needing to use recursion to fetch the subjects from the database. You could add a root ForeignKey to Subject on Subject that would identify the root subject. This would allow you to grab all subjects in a tree fairly easily. Then you'd have to arrange them in your View/Viewset to fit the ordering (if that's necessary).
  2. Use raw() and your database's recursive functionality to fetch the models. This would require raw SQL and can be painful to maintain.
  3. Use django_cte. I've used this in one of my projects for a few queries, but I'm not a big fan of it. It breaks some functionality with update() being called on an empty queryset. However, it will work and it won't require you to drop down to raw SQL.

Upvotes: 1

Related Questions