Django rendering dict performance

Question

Hi, I have a somewhat complex dict that I want to render. Rendering it now takes about 10-15 seconds which is clearly undesirable.

Currently I have a dict with 237 keys (It may be as little as three, and as much as about 700).

The SQL is done manually (cur.execute and cur.fetchall()), so nothing lazy should happen in the template itself (Which the debug toolbar also agrees on). The profiling shows that it is pure rendering which is slow, even if I do not draw this large dict anywhere in my template. If I omit passing it at all to the render() function everything is as snappy as it should be.

What on earth should I do to make this more speedy?

The dict with one element of 200+ looks somewhat like this:

data[entry_id] ={ 
{
'status': 200,
'statustext': u'OK',
'rhttpversion': u'HTTP',
'host': u'l.linkpulse.com',
'entry_id': 16484,
'port': None,
'uri': u'http: //l.linkpulse.com/p.gif',
'response_header': [
    {
        'name': u'Connection',
        'value': u'keep-alive'
    },
    {
        'name': u'Content-Length',
        'value': u'43'
    },
    {
        'name': u'Server',
        'value': u'Apache/2.2.14(Ubuntu)'
    },
    {
        'name': u'Last-Modified',
        'value': u'Mon,
        07May200711: 49: 44GMT'
    },
    {
        'name': u'ETag',
        'value': u'"8607e-2b-42fe5a7"'
    },
    {
        'name': u'Date',
        'value': u'Sat,
        09Nov201317: 34: 40GMT'
    },
    {
        'name': u'Content-Type',
        'value': u'image/gif'
    },
    {
        'name': u'Accept-Ranges',
        'value': u'bytes'
    }
],
'request_header': [
    {
        'name': u'Referer',
        'value': u'http://www.xxx.com/'
    },
    {
        'name': u'Accept-Language',
        'value': u'en-US,
        en;q=0.5'
    },
    {
        'name': u'Accept',
        'value': u'text/html,
        application/xhtml+xml,
        application/xml;q=0.9,
        */*;q=0.8'
    },
    {
        'name': u'Host',
        'value': u'l.linkpulse.com'
    },
    {
        'name': u'User-Agent',
        'value': u'Mozilla/5.0(Windows;U;WindowsNT6.1;en-US;rv: 24.0)Gecko/20100101Firefox/24.0'
    },
    {
        'name': u'Accept-Encoding',
        'value': u''
    }
],
'method': u'GET',
'httpversion': u'HTTP'
}
}

EDIT: sys.sizeof(data) is 280. Do note that each of the elements in the troublesome dict have two arrays with an average of 6-7 dicts (with two key/value pairs each).

The template looks something like this:

{% extends "layouts/full.html" %}
{% block content %}
...
{% include 'report/headers.html' %}
...

report/headers.html:

<table class="table table-striped table-bordered table-hover headers">
  <thead>
    <th class="hheader" style="text-align:center">Request</th>
    <th class="hheader" style="text-align:center">Response</th>
  </thead>
  <tbody>
    {% for key,header in headers.items %}
    <tr class="connection">
      <td class="headers">
        <b>
          {{ header.method }} {{ header.uri }} {{ header.rhttpversion }}
          <br>
          <br>
        </b>
        <div class="headers">
          {% for request in header.request_header %}
          <b> {{ request.name }}: </b> {{ request.value }} <br>
          {% endfor %}
        </div>
      </td>
      <td class="headers">
        <b>
          {{ header.status }} {{ header.statustext }} {{ header.httpversion }}
          <br>
          <br>
        </b>
        <div class="headers">
          {% for response in header.response_header %}
          <b> {{ response.name }}: </b> {{ response.value }} <br>
          {% endfor %}
        </div>
        {% endfor %}
      </td>
    </tr>
  </tbody>
</table>

Relevant parts from the profiling: Template timings: https://www.dropbox.com/s/rlqym0akg45sur1/Screenshot%202013-11-10%2019.03.36.png Time: https://www.dropbox.com/s/lkuwfsb8sygi1es/Screenshot%202013-11-10%2019.20.49.png

Ill update soon-ish with info from the Profile tab. It is for some reason greyed out at the moment.

The site with the data can be seen here: http://two.mrfjo.org/1316c6f4-72c1-46b6-818c-31121d2c9712/ (Disregard the snort alerts, they arent real).

Answer 1

I worked around this problem by generating the HTML myself in the view.

def headers_html(headers):
    html = ""
    for key,header in headers.items():
        html+="""<tr class="connection">
        <td class="headers">
          <b>"""
        html += header['method'] + " " + header['uri'] + " " + header['rhttpversion']
        html +="""<br>
            <br>
          </b>
          <div class="headers">"""
        for request_headers in header['request_header']:
            html +="<b> {}: </b>{} <br>".format(request_headers['name'],request_headers['value'])
        html += """</div>
        </td>
        <td class="headers">
          <b>"""
        html += "{} {} {}".format(header['status'], header['statustext'], header['httpversion'])
        html +="""<br>
            <br>
          </b>
          <div class="headers">"""
        for response in header['response_header']:
            html +="""<b>{}: </b> {} <br>""".format(response['name'], response['value'])
        html +="""</div>"""

    return html

And in the template:

{% autoescape off %}{{ headers }}{% endautoescape %}

Answer 2

Django debug toolbar is incredibly useful, but I find it does slow down page loads for complex pages.

Try disabling the toolbar and see if it improves the performance.