CODEWITHSUNDEEP
pythondjangodatabasesqliterender
pythonize
pythonize

Reputation: 71

Very slow django with "large" database (< 1M rows)

I'm trying to build an app with django. When I work with a small database (around 10k rows), the app works perfectly. However, in some views, I need to calculate some data of my classes, so I need to call all of my database. It is very slow (more than 30s), and according to Django Debug Toolbar, most part of the time is consumed by CPU.

Here's an example of a (very) simplified code I try to launch:

def test_view(request):
    queryset_articles = Articles.objects.all()
    total_count = sum([a.words_count for a in queryset_articles])
    response = 'Total words : ' + str(count)
    return HttpResponse(response)

How should I do to get a reasonable waiting time? I think about doing another table, which I would update frequently with these historical stats, then call them directly in my view.

Thank you,

Pierre

Upvotes: 0

Views: 1248

Answers (3)

pythonize
pythonize

Reputation: 71

Thank you for you answers, this view is now a lot faster! However, I got some views to calculate average with different filters, like this:

        hours_of_weekday = [Article.objects.filter(
            date__hour=k,
            date__week_day=weekday_num,
            type__in=categories).aggregate(Avg('count'))['count__avg'] for k in range(0, 24)]

Which is still very slow with this... Any idea except making a table dedicated to these values?

Upvotes: 0

Mickaël Martinez
Mickaël Martinez

Reputation: 1843

First of all, you should check if the Django Debug Toolbar itself is not the reason for this waiting time (its documentation explains the problem and provides some solutions here).

Then, did you try to use Aggregation ? This could optimize your code:

from django.db.models import Sum

Articles.objects.aggregate(Sum('words_count'))

Upvotes: 1

Chillie
Chillie

Reputation: 1485

Try aggregation. It can provide a small to noticeable increase in time since it uses a database query to count.

from django.db.models import Sum

total_count = Articles.objects.all().aggregate(Sum('words_count'))
print(total_count)

Upvotes: 1

Related Questions