CODEWITHSUNDEEP
djangomongodbpaginationpymongo
Dor
Dor

Reputation: 902

Using Django's Paginator class with a MongoDB cursor

Is there a known method of extending the MongoDB cursor object to make it compatible with Django's django.core.paginator.Paginator class?

Or, maybe, extend Django's class?

Upvotes: 3

Views: 2246

Answers (3)

womblerone
womblerone

Reputation: 532

The approach from @cpury didn't work for me - maybe there were some pymongo/django changes. Here is a simpler version that creates a paginator-like object for the ListView to interact with. Note that in your template you'll need to use page_obj for attributes like has_other_pages, has_next, has_previous etc.

class MongoQuerySet:
    def __init__(self, cursor, results_count):
        self.cursor = cursor
        self.count = results_count

    def __getitem__(self, key):
        if isinstance(key, slice):
            return list(self.cursor[key.start:key.stop])
        else:
            return self.cursor[key]

    def __len__(self):
        return self.count

class MongoSearchView(LoginRequiredMixin, ListView):
    template_name = 'pages/results.html'
    context_object_name = 'results'
    paginate_by = 15

    def get_queryset(self):
        client = MongoClient(
            host=settings.MONGO_HOST,
            port=settings.MONGO_PORT,
            username=settings.MONGO_USER,
            password=settings.MONGO_PASSWORD,
            authSource=settings.MONGO_AUTH_SOURCE
        )
        db = client[settings.MONGO_DB]
        collection = db[settings.MONGO_COLL]
        index_info = collection.index_information()
        if "filename_text" not in index_info:
            collection.create_index([("filename", "text")])

        query = self.request.GET.get('q', '')
        regex_pattern = f".*{re.escape(query)}.*"
        projection = {"filename": 1, "description": 1, "created": 1}
        filter_dict = {"filename": {"$regex": regex_pattern, "$options": "i"}}
        results = collection.find(filter_dict, projection)
        results_count = collection.count_documents(filter_dict)


        return MongoQuerySet(results, results_count)

Upvotes: 0

cpury
cpury

Reputation: 1023

I was facing the same issue and implemented my own Paginator class that works. Here's the code:

from django.core.paginator import Paginator, Page

class MongoPaginator(Paginator):
    """
    Custom subclass of Django's Paginator to work with Mongo cursors.
    """
    def _get_page(self, *args, **kwargs):
        """
        Returns an instance of a single page. Replaced with our custom
        MongoPage class.
        """
        return MongoPage(*args, **kwargs)

    def page(self, number):
        """
        Returns a Page object for the given 1-based page number.
        Important difference to standard Paginator: Creates a clone of the
        cursor so we can get multiple slices.
        """
        number = self.validate_number(number)
        bottom = (number - 1) * self.per_page
        top = bottom + self.per_page
        if top + self.orphans >= self.count:
            top = self.count
        return self._get_page(
            self.object_list.clone()[bottom:top], number, self
        )

class MongoPage(Page):
    """
    Custom Page class for our MongoPaginator. Just makes sure the cursor is
    directly converted to list so that we can use len(object_list).
    """
    def __init__(self, object_list, number, paginator):
        self.object_list = list(object_list)
        self.number = number
        self.paginator = paginator

The main changes are:

  • Each Page should get a clone of the cursor, because slices work only once per cursor
  • Each Page should also directly convert it to a list so that len() works

Here is a helper function to use it in views:

def get_paginated_cursor(request, cursor, per_page=25, param='page'):
    """
    Helper to deal with some standard pagination. Pass a request and a
    cursor and it will return a paginated version of it.
    """
    paginator = MongoPaginator(cursor, per_page)

    page = request.GET.get(param, 1)
    try:
        cursor = paginator.page(page)
    except PageNotAnInteger:
        # If page is not an integer, deliver first page.
        cursor = paginator.page(1)
    except EmptyPage:
        # If page is out of range (e.g. 9999), deliver last page of results.
        cursor = paginator.page(paginator.num_pages)

    return cursor

Then you can do:

def some_view_function(request):
    messages_cursor = db.messages.find({})
    messages = get_paginated_cursor(request, messages_cursor)

Upvotes: 1

mpobrien
mpobrien

Reputation: 4972

Your temporary solution (https://gist.github.com/2351079) looks good - but instead of forcing the cursor to fetch all results with list() and paginating with [bottom:top], maybe try using .skip() and .limit() on the cursor explicitly - it will probably improve performance.

Upvotes: 0

Related Questions