CODEWITHSUNDEEP
pythonmysqldjangodatabasepagination
E.Diaz
E.Diaz

Reputation: 105

Django Pagination too slow with large dataset

I have a problem regarding Django's Pagination. When I try to paginate over 200k records the portal Web load very slow (>10 secs), and I'm suppose to use around 2 million records.

I can't find on Stackoverflow or other site a GOOD specific fix to this problem. Everytime the code gets a page it executes the QuerySet, which goes over a very large dataset, making it slow.

Anybody knows what can be done? I've searched everywhere and couldn't fix this issue. Below I post the code to paginate. Paper_list is: model.object.all().filter(category=x) (and there are around 200k objects now (all of them belong to that category too).

def paginate_context(paper_list, request, context, source, author, title, term, date):
num_papers = len(paper_list) 
paginator = Paginator(paper_list, 4) # Show X papers per page
page = request.GET.get('page')
try:
    papers = paginator.page(page)
except PageNotAnInteger:
    # If page is not an integer, deliver first page.
    papers = paginator.page(1)
except EmptyPage:
    # If page is out of range (e.g. 9999), deliver last page of results.
    papers = paginator.page(paginator.num_pages)      
context['papers'] = papers
context['num_papers'] = num_papers
context['query_cat'] = create_request_str_and(context['cat'], source, author, title, term, date)

Upvotes: 9

Views: 7447

Answers (1)

ruddra
ruddra

Reputation: 51948

As I can see in the code, num_papers = len(paper_list) method evaluates the query, hence could hamper performance. You can change it to:

num_papers = paper_list.count()

You can check here about when the queryset is evaluated: https://docs.djangoproject.com/en/1.9/ref/models/querysets/#when-querysets-are-evaluated

Upvotes: 4

Related Questions