Django: Paginate by sorted SerializerMethodField

Question

I'm having an issue with paginating json data that is sorted by the rest_framework's SerializerMethodField. Before I began to add pagination to my list views, I had the sorted json data in a context variable like this:

class ExampleList(ListView):
    ...
    def get_context_data(self, **kwargs):
        context = super(ExampleList, self).get_context_data(**kwargs)
        context["examples"] = sorted(ExampleSerializer(
            submissions, many=True, context={'request': self.request}
        ).data, key=lambda x: x.get("score"), reverse=True)
        return context
    ...

This worked perfectly, because the lambda function grabbed the score, and ordered by it, exactly how sorted() should work. The problem began with pagination. I've researched for a few days now, and there aren't any methods to paginate by json data that I can find. Only by querysets.

When I started paginating, here are my two Serializer Classes:

class ExampleSerializer(serializers.ModelSerializer):
    score = serializers.SerializerMethodField('get_score')

    class Meta:
        model = Example
        fields = ('id', '...', 'score',)

    def get_score(self, obj):
        return obj.calculate_score()

class PaginatedExampleSerializer(pagination.PaginationSerializer):
    class Meta:
        object_serializer_class = ExampleSerializer

In one of my list views, I've created a sorted context object that sorts the serialized data by score and paginates it. I, also, created a method to call for pagination called paginate_examples(). As you can see, it paginates by the queryset first, and then sorts the data by score on each paginated page. So something that should be on page 1 is all the way back on page 5 or so.

class ExampleList(ListView):
    queryset = Example.objects.all()

    def paginate_examples(self, queryset, paginate_by):
        paginator = Paginator(queryset, paginate_by)
        page = self.request.GET.get('page')
        try:
            examples = paginator.page(page)
        except PageNotAnInteger:
            examples = paginator.page(1)
        except EmptyPage:
            examples = paginator.page(paginator.num_pages)

        return PaginatedExampleSerializer(examples, context={'request': self.request}).data

    def get_context_data(self, **kwargs):
        context = super(ExampleList, self).get_context_data(**kwargs)

        pagination = self.paginate_examples(self.queryset, self.paginate_by)
        examples = pagination.get("results")
        context["examples"] = sorted(examples, key=lambda x: x.get("score"), reverse=True)
        context["pagination"] = pagination
        return context

Again, the problem is that list items that should be displaying on /?page=1 are displaying on /?page=x because the PaginatedExampleSerializer paginates the data before it gets sorted by the SerializerMethodField.

Is there a way to paginate data that is already serialized, instead of paginating by the queryset in Django? Or am I going to have to create some methods myself? I would like to avoid making score a database field, but if I can't figure out a solution, then I guess I will have to. Any help on this would be greatly appreciated.

Answer 1

Kind of a late answer, but might help someone who finds themselves here.

I had a similar issue where I needed to combine two separate models in the output, picking the last item among the two models, with pagination, and ordering. I found that OrdereringFilter was a bit too much for what I was looking for, so opted to default the ordering, however this should work with the OrderingFilter as well.

The basic approach is to determine a way via database functions and aggregate function to annotate the field calculated by the serializer.

from rest_framework.pagination import LimitOffsetPagination
from rest_framework.generics import ListAPIView
from django.db.models import Q, Count, Avg
from django.db.models.functions import Greatest

class ExampleListViewPagination(LimitOffsetPagination):
    default_limit = 10

class ExampleSerializer(serializers.ModelSerializer):
    
    def get_last_message_and_score(self, obj):
        # added to obj via the view get_queryset annotate
        return {
          "last_message": self.get_last_comment_or_file(),
          "last_message_time": obj.last_message_time,
          "score": obj.score
         }

    class Meta:
        model = Example
        fields = ('id', '...', 'last_message_and_score',)

class ExampleList(ListAPIView):
    serializer_class = ExampleSerializer
    pagination_class = ExampleListViewPagination

    def get_queryset(self):
        # the annotation here is being used for the pagination to work
        # on last_message_time, and hypothetical calculate_score
        # both descending
        qs = (
            Example.objects.filter(
                Q(item__parent__users=self.get_user())
                & (
                    Q(last_read_by_user=None)
                    | Q(last_read_by_user__lte=timezone.now())
                )
            )
            .exclude(
                Q(user_comments__isnull=True) 
                & Q(user_files__user_item_files__isnull=True)
            )
            .annotate(
                last_message_time=Greatest(
                    "user_comments__created",
                    "user_files__user_item_files__created",
                ),
                score=Avg(Count("user_comments"), Count("user_files__user_item_files")),
                )
            )
            .distinct()
            .order_by("-last_message_time", "-score")
        )
        return qs

One of the caveats here is that prefetch_related, and select_related might be able to optimize the query a bit.

One of the other options I tried exploring was based on, overriding paginate_queryset on rest_framework.pagination.LimitOffsetPagination to do the pagination outside of the database query and queryset, but found the annotation easier.