subprocess blocks Django view

Question

I have a problem calling subprocess.Popen from a view: The view that calls subprocess.Popen is not displayed until the subprocess finishes. The server send "200 OK" immediately, but not the content of the page.

My question is: Is this a limitation of Django's development server or am I doing it wrong?

The server does not completely hangs, as other views can be processed in the meantime.

There are already a few questions on that topic and Google gives a few other threads, but I cannot find a clear answer to my question.

I believe this is not a python issue as this commands terminate immediately:

python -c 'import subprocess; print subprocess.Popen(["/bin/sleep", "10"]).pid'

How to reproduce

Create test project & app:

cd /tmp
django-admin.py startproject django_test
cd django_test
./manage.py startapp subprocess_test

Replace urls.py & subprocess_test/views.py with:

• urls.py:

  from django.conf.urls.defaults import *

  urlpatterns = patterns('',
    (r'^hello$', 'subprocess_test.views.hello'),
    (r'^start$', 'subprocess_test.views.start'),
  )

• subprocess_test/views.py

  from django.http import HttpResponse

  import subprocess

  def hello(request):
    return HttpResponse('Hello world!')

  def start(request):
    subprocess.Popen(["/bin/sleep", "10"])
    return HttpResponse('start done')

Test it:

./manage.py runserver 0.0.0.0:8000

Go to http://127.0.0.1:8000/hello and http://127.0.0.1:8000/start

Test result

"start" takes 10s to load and "hello" can be loaded during that time. For example, I get such a log:

[01/Feb/2011 07:20:57] "GET /hello HTTP/1.1" 200 12
[01/Feb/2011 07:21:01] "GET /start HTTP/1.1" 200 10
[01/Feb/2011 07:21:01] "GET /hello HTTP/1.1" 200 12
[01/Feb/2011 07:21:02] "GET /hello HTTP/1.1" 200 12

Using wget:

wget http://127.0.0.1:8000/start
--2011-02-01 14:31:11-- http://127.0.0.1:8000/start
Connecting to 127.0.0.1:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `start'

[           <=>                           ] 10          --.-K/s   in 9,5s    

2011-02-01 14:31:21 (1,05 B/s) - « start » saved [10]

Answer 1

Try this

import subprocess

x = subprocess.Popen(["/bin/sleep", "10"])
x.wait()

Answer 2

I encountered the same problem running Django with nginx+uwsgi. Often the breaking point is the module of web-server, in my case it was uwsgi. Adding <close-on-exec/> solved this problem. From other side the fastcgi module doesn't cause to view to hung on background processes.

I don't know which server do you use, so you can check this behavior with various modules (uwsgi, mod_wsgi, fastcgi) and see what is more suitable for you.
And try to execute in background:

subprocess.Popen(["/bin/sleep", "10", "&"])

Answer 3

It looks like you don't care what the result of the system call is, so I would assume you are trying to do some sort of offline(or background) processing.

I would suggest a cleaner way of going about it rather than directly executing a program. Use a queueing system such as Gearman to queue up processing tasks and then have a separate worker that consumes items from the queue.

This has the advantage of protecting your server when large traffic spikes happen, so you don't fork off a process each time a request to that view is made. You can consume items as slow or as fast as you decide, independent of traffic.

Traffic may not be an issue, but I personally think it is a cleaner design decision as well.