CODEWITHSUNDEEP
pythondjangopaginator
Kelp
Kelp

Reputation: 1268

How to optimize for Django's paginator module

I have a question about how Django's paginator module works and how to optimize it. I have a list of around 300 items from information that I get from different APIs on the internet. I am using Django's paginator module to display the list for my visitors, 10 items at a time. The pagination does not work as well as I want it to. It seems that the paginator has to get all 300 items before pulling out the ten that need to be displayed each time the page is changed. For example, if there are 30 pages, then going to page 2 requires my website to query the APIs again, put all the information in a list, and then access the ten that the visitor's browser requests. I do not want to keep querying the APIs for the same information that I already have on each page turn.

Right now, my views has a function that looks at the get request and queries the APIs for information based on the query. Then it puts all that information into a list and passes it onto the template file. So, this function always loads whenever someone turns the page, resulting in querying the APIs again.

How should I fix this?

Thank you for your help.

Upvotes: 2

Views: 1342

Answers (2)

nate c
nate c

Reputation: 9005

ORM's do not load data until the row is selected:

query_results = Foo(id=1) # No sql executed yet, just stored.

foo = query_results[0] # now it fires

or

for foo in query_results:
   foo.bar() # sql fires

If you are using a custom data source that is loading results on initialization then the pagination will not work as expected since all feeds will be fetched at once. You may want to subclass __getitem__ or __iter__ to do the actual fetch. It will then coincide with the way Django expects the results to be loaded.

Pagination is going to need to know how many results there are to do things like has_next(). In sql it is usually inexpensive to get a count(*) with an index. So you would also, want to have know how many results there would be (or maybe just estimate if it too expensive to know exactly).

Upvotes: 0

Marcus Whybrow
Marcus Whybrow

Reputation: 19998

The paginator will in this case need the full list in order to do its job.

My advice would be to update a cache of the feeds at a regular interval, and then use that cache as the input to the paginator module. Doing an intensive or length task on each and every request is always a bad idea. If not for the page load times the user will experience, think of the vulnerability of your server to attack.

You may want to check out Django's low level cache API which would allow you to store the feed result in a globally accessible place under a key, which you can later use to retrieve the cache and paginate for each page request.

Upvotes: 1

Related Questions