CODEWITHSUNDEEP
pythondjangoperformancewsgigunicorn
Ward
Ward

Reputation: 97

Django website performance slowdown

All righty so I want to explain my small django issue, that I am having trouble getting around.

The Problem

I have a small website, just a couple of pages that display a list of database records. The website is an internal render farm monitor for my company which will have perhaps a dozen or two active connections at any time. No more than 50.

The problem is that I have three update services that cause a real performance hit when turned on.

The update services each are python scripts that:

  1. Use urllib2 to make a http request to a url.
  2. Wait for the response
  3. Print a success message with time stamps to a log.
  4. Wait 10 seconds, and start again.

The URLs they send requests to cause my django website to poll an external service and read new data into our django database. The urls look like this:

When these update services are turned on (especially updateTasks), it can take well over 10 seconds for http://webgrid/ to even start loading for normal users.

The Setup

Django 1.8, deployed with Gunicron v18.

The main gunicorn service is run with these arguments (Split into a list for easier reading).

<PATH_TO_PYTHON>
<PATH_TO_GUNICORN> 
-b localhost:80001 
-u farmer 
-t 600 
-g <COMPANY_NAME> 
--max-requests 10000 
-n bb_webgrid 
-w 17 
-p /var/run/gunicorn_bb_webgrid.pid 
-D 
--log-file /xfs/GridEngine/bbgrid_log/bb_webgrid.log
bb_webgrid.wsgi:application

Apache config for this site:

<VirtualHost *:80>
    ServerName webgrid.<INTERAL_COMPANY_URL>
    ServerAlias webgrid

    SetEnv force-proxy-request-1.0 1

    DocumentRoot /xfs/GridEngine/bb_webgrid/www
    CustomLog logs/webgrid_access.log combined
    ErrorLog logs/webgrid_error.log
    #LogLevel       warn
    <Directory "/xfs/GridEngine/bb_webgrid/www">
            AllowOverride All
    </Directory>

    WSGIDaemonProcess webgrid processes=17 threads=17
    WSGIProcessGroup webgrid

</VirtualHost>

Upvotes: 0

Views: 170

Answers (1)

Daniel Roseman
Daniel Roseman

Reputation: 599450

This kind of thing shouldn't be done online; by hitting a URL which directs to a view you are unnecessarily tying up your webserver which stops it from doing its real job, which is to respond to user requests.

Instead, do this out-of-band. A really quick an easy way to do this is to write a Django management command; that way you can easily call model methods from a command-line script. Now you can simply point your cron job, or whatever it is, to call these commands, rather than calling a separate Python script which calls a URL on your site.

An alternative is to use Celery; it's a really good system for doing long-running asynchronous tasks. It even has its own scheduling system, so you could replace your cron jobs completely.

Upvotes: 4

Related Questions