CODEWITHSUNDEEP
djangodjango-models
user193130
user193130

Reputation: 8237

Django: Filtering a queryset locally from cache

If I perform a prefetch_related('toppings') for a queryset, and I want to later filter(spicy=True) by fields in the related table, Django ignores the cached info and does a database query. I found that this is documented (under the Note box) and seems to happen for all forms of caching (select_related(), already evaluated querysets, etc.) when another filter() is performed.

However, is there some sort of super secret hidden time-saving shortcut to filter locally (using the cache and not hitting the database) without having to write the python code to loop the queryset (using list/dict comprehension, etc.)? Maybe something like a filter_locally(spicy=True)?

EDIT:

One of the reasons why a list/comprehension doesn't work well for me is because a list/dict does not have the queryset methods. In my case, the first level M2M field, toppings, isn't the end goal for me and I need to check a 2nd related M2M field (which I have already pre-fetched as well). While this is also possible using list comprehension, it's just much simpler to have something such as filter_locally(spicy=True, origin__country='Spain') because:

  1. it allows accessing many levels of related fields with minimal effort
  2. it allows chaining other queryset methods
  3. it's easier to read because it's consistent with the familiar filter()
  4. it's easier to modify existing code using filter() without prefetch to add this optimization in without much changes.

But from the responses, Django has no such support :(

Upvotes: 7

Views: 5365

Answers (3)

Vlad Marchenko
Vlad Marchenko

Reputation: 272

Unfortunately, still not possible :( .filter() always make query to db

for simple cases this should work:

def filter_queryset(queryset, **kwargs):
    res = []
    for instance in queryset:
        if all((instance.__getattribute__(field)==value for field, value in kwargs.items())):
            res.append(instance)
    return res

#and then
#instead of User.objects.filter(age=25, sex="Male")

filter_queryset(User.objects.all(), age=25, sex="Male")

but this approach doesnt have features like .id__in = some_list.

Its hard to make custom args processor so maybe something like this is possible:

def filter_queryset(queryset, *args):
    res = []
    for instance in queryset:
        if all((func(instance) for func in args)):
            res.append(instance)
    return res

#and then
#instead of User.objects.filter(id__in=some_list, age__gt=25)

filter_queryset(User.objects.all(),
                lamda inst: inst.id in some_list,
                lamda inst: inst.age > 25)

looks kinda funny but works.

Upvotes: 0

Ian weisberger
Ian weisberger

Reputation: 194

If you're filtering on a boolean doing the list comprehension is pretty easy. You can also swap out the topping.spicy==True for a string comparison or whatever.

I would do something like:

qs = Pizza.objects.all().prefetch_related('toppings')
res = list(qs)

def get_spicy(qs):
    res = list(qs)
    return [pizza  for pizza in res if any(topping.spicy==True for 
                                topping in pizza.toppings.all())]

That is if you want to return the pizza object if any of its toppings is spicy. You can also replace the any() with all() to check for all, and do a lot of pretty powerful queries with this syntax. I'm somewhat surprised that there is no easy way to do this in django. It seems like a lot of these simple queries should be easy to implement in a generic manner.

The above code assumes a many2many. It should be easy to modify to work with a simple FK relationship such as a one2one or one2many.

Hope this was helpful.

Upvotes: 2

Andrew Gorcester
Andrew Gorcester

Reputation: 19983

You have to write the python code to loop through the queryset (a list/dict comprehension is ideal). All the filter() code knows how to do is add filtering language to the SQL sent to the database. Filtering locally is a totally different problem than filtering remotely, so the solutions to those two separate problems won't be able to share any logic.

A list comprehension one-liner would be pretty straightforward, though; the syntax might not be much more complex than with filter().

Upvotes: 3

Related Questions