CODEWITHSUNDEEP
djangooptimizationdjango-templates
laszchamachla
laszchamachla

Reputation: 773

Django - rendering many templates using templatetags is very slow

Say, I have a page with a photo gallery. Each thumbnail has e.g. a photo, country, author and so on. I render these items/widgets using template tags (which load specified templates) - it goes that way because of DRY (I use these items/widgets separately in different places on the page).

And it is very slow.

I have performed some profiling using django-debug-toolbar:

SQL Queries: default 84.81 ms (147 queries)

But:

Total CPU time: 5768.360 msec

Which is too long to wait.

After some analysis it turned out that the main culprit is templating enginge.

When I want to display e.g. 150 photos, 600 associated items/widgets are being rendered via templates. It means 600 I/O operations or even more. Moving these widgets to main template solves the problem, but does not keep DRY.

So my question is how one can avoid such a behaviour? Be DRY and slow or no-DRY and fast? I'd rather be DRY and fast...

Upvotes: 16

Views: 7841

Answers (6)

Arthur C
Arthur C

Reputation: 1274

I had the same issue, and maybe for the same reason. I optimized the performance of a custom template tag. The number of db requests fell from 640 to 2, and the resulting db time was under 20ms. My page however, had become slower! 7s -> 10s. Sigh. I tried a cached template loader, without effect.

I had given up, disabled the django debug toolbar, after which the response time fell to 1.2s, unbelievable!! In my case, the huge response time was only caused by the debug toolbar! A related issue can be found here. I'm not diving deeper in this toolbar issue, as I was planning to cache the template tag using template fragment caching anyways. During development I have 10s response time every 15 minutes, which I can live with. Anyways, hope this helps.

Upvotes: 1

Ivan Virabyan
Ivan Virabyan

Reputation: 1686

Time spent on queries is relatively little. But ORM takes much more time to generate these queries and parse the results into the model instances.

So, despite huge number of queries your app is CPU bound because of the slow ORM (in your case it may take a second). So you have to reduce number of queries anyway.

Most probably the queries are made inside of your template tag. So you have to get desired data in a few queries, and set it on the photo instances.

{% for photo in photos|annotate_comment_count %}
   ...
{% endfor %}

def annotate_comment_count(photo_list):
    counts = dict(Comment.objects.filter(photo__in=photo_list).values('photo') \
                                 .annotate(count=models.Count('id')))
    for photo in photo_list:
        photo.comments_count = counts[photo.pk]
    return photo_list

So, inside your templatetag you don't have to query comments count for a single photo, but you already have this information in comments_count attribute. And you achieved this in one query.

Upvotes: 0

laszchamachla
laszchamachla

Reputation: 773

After several hours of profiling and searching...

Thanks for your help, but in this case it seems to me that the best solution so far is to use Template fragment caching:

I tried it and gained 70-80% speed performance!

{% load cache %}
{% cache 3600 mywidget_id %}
    .. rendered mywidget is cached ..
{% endcache %}

Upvotes: 11

Virgiliu
Virgiliu

Reputation: 3118

This might not apply to this particular problem but in some cases it helped me to use select_related on the queries. Might not be the case but it might lower your query count.

Upvotes: 0

Chris Pratt
Chris Pratt

Reputation: 239290

I'm assuming since you're using the debug toolbar that you're getting these numbers in development. However, because of this, these are not "real" numbers.

The built-in Django server is good for development, but it has a number of shortcomings that make it much slower that a real webserver would be. First, it's single threaded, so that means no parallel requests. This also means that IO ops are discrete. Second, it's tasked with not just serving requests to Django, but also static resources.

Long and short, if you want to truly profile your site for page load times, you'll need to install a true webserver locally. Basically set it up like you would in your production environment. I'd be willing to wager the request times will be far better, then.

Upvotes: 4

Daniel Roseman
Daniel Roseman

Reputation: 599610

You might want to try the caching template loader, django.template.loaders.cached.Loader - it should certainly reduce the amount of IO needed.

Edit to add You need to be careful of assuming that just because the majority of time is spent in the template rendering phase, that the query count is not to blame. Don't forget that querysets are lazy, and unless you're specifically slicing or iterating them in the view, they will only be evaluated when the template is loaded. I would say that reducing your query count through good use of select_related and other techniques should be significant help.

Upvotes: 10

Related Questions