CODEWITHSUNDEEP
pythondjangodjango-queryset
John A. Vink
John A. Vink

Reputation: 301

Iterating over Django QuerySet without pre-populating cache

I want to iterate over all the objects in a QuerySet. However, the QuerySet matches hundreds of thousands if not millions of objects. So when I try to start an iteration, my CPU usage goes to 100%, and all my memory fills up, and then things crash. This happens before the first item is returned:

bts = Backtrace.objects.all()
for bt in bts:
  print bt

I can ask for an individual object and it returns immediately:

bts = Backtrace.objects.all()
print(bts[5])

But getting a count of all objects crashes just as above, so I can't iterate using this method since I don't know how many objects there will be.

What's a way to iterate without causing the whole result to get pre-cached?

Upvotes: 0

Views: 1315

Answers (1)

raratiru
raratiru

Reputation: 9616

First of all make sure that you understand when a queryset is evaluated (hits the database).

Of course, one approach would be to filter the queryset.

If this is out of the question there are some workarounds you can use, according to your needs:

Here is a nice article that tries to tackle with this issue in a more theoretical level.

Upvotes: 3

Related Questions