CODEWITHSUNDEEP
pythondjango
tback
tback

Reputation: 11571

How to detect deadlocks in django applications (and get rid of them)

I am maintaining a django project that gets unresponsive on regular basis. So far I dealt with the situation by continuously monitoring the application and restarting apache when necessary.

Unresponsive how? It means that apache does not reply to any request anymore.

The environment:

What can I do to get closer to the root of the error? (I cannot not provide the source code - snippets here and there are possible though) I've hunted this problem for so long that it is impossible to list all the things I tried. I tried to get rid of any 'magic' that I could think of. Several parts of the application have been rewritten since the problem occurred.

I am sorry for the lack of detail, but I'll happily provide (almost) any information that is requested and promise to do my best to make this post as helpful as possible to others facing similar problems.

Upvotes: 1

Views: 2658

Answers (2)

harmv
harmv

Reputation: 1922

You might be bitten by the following django bug [1] (that is not yet fixed in 1.4 branch)

workaround: manuall apply the fix to your django sources, or use a threadsafe wrapper around the the wsgi module as shown below (we use this in production systems)

from __future__ import with_statement
from  django.core.handlers.wsgi import WSGIHandler as DjangoWSGIHandler

from threading import Lock

__copyright__ = "Jibe"

class WSGIHandler(DjangoWSGIHandler):
    """
    This provides a threadsafe drop-in replacement of django's WSGIHandler.

    Initialisation of django via a multithreaded wsgi handler is not safe.
    It is vulnerable to a A-B B-A deadlock.

When two threads bootstrap django via different urls you have a change to hit 
the following deadlock.

  thread 1                                               thread  2
    view A                                                  view B
     import file foo            import lock foo               import file bar  import lock bar
           bootstrap django     lock AppCache.write_lock
                import file bar import lock bar  <-- blocks
                                                                 bootstrap django    lock AppCache.write_lock  <----- deadlock

workaround for an AB BA deadlock:  wrap it in a lock C.

        lock C                      lock C
            lock A                      lock B
            lock B                      lock A
            release B                   release A
            release A                   release A
        release C                   release C          

    Thats exactly what this class does,  but... only for the first few calls.  
    After that we remove the lock C.  as the AppCache.write_lock is only held when django is booted. 

    If we would not remove the lock C after the first few calls, that would make the whole app single threaded again. 

    Usage:    
        in your wsgi file replace   the following lines 
                import django.core.handlers.wsgi.WSGIHandler  
                application = django.core.handlers.wsgi.WSGIHandler 
        by 
                import threadsafe_wsgi 
                application = threadsafe_wsgi.WSGIHandler 


    FAQ: 
        Q: why would you want threading in the first place ?                 
        A: to reduce memory. Big apps can consume hundeds of megabytes each.  adding processes is then much more expensive than threads. 
           that memory is better spend caching, when threads are almost free. 

        Q: this deadlock, it looks far-fetched, is this real ? 
        A: yes we had this problem on production machines. 
    """ 
    __initLock = Lock()  # lock C 
    __initialized = 0 

    def __call__(self, environ, start_response): 
        # the first calls (4) we squeeze everybody through lock C 
        # this basically serializes all threads 
        MIN_INIT_CALLS = 4 
        if self.__initialized < MIN_INIT_CALLS: 
            with self.__initLock: 
                ret = DjangoWSGIHandler.__call__(self, environ, start_response) 
                self.__initialized += 1 
                return ret 
        else: 
            # we are safely bootrapped, skip lock C 
            # now we are running multi-threaded again 
            return  DjangoWSGIHandler.__call__(self, environ, start_response)

and in your wsgi.py use the following code

from threadsafe_wsgi.handlers import WSGIHandler
django_handler = WSGIHandler()

[1] https://code.djangoproject.com/ticket/18251

Upvotes: 1

Graham Dumpleton
Graham Dumpleton

Reputation: 58563

Ultimately you need the new features that have been added to mod_wsgi 4.0. These will allow for daemon mode better control over automatic restarts when requests block. When restarting upon blocked condition mod_wsgi will attempt to dump out Python stack traces for what each of the Python request threads was doing at the time so you can see why they are blocked.

Suggest you take up the issue on the mod_wsgi mailing list and can explain the new features in more detail if need be. Have posted before about it at:

http://groups.google.com/group/modwsgi/msg/2a968d820e18e97d

The mod_wsgi 4.0 code is only available from source code repository at this time. Current trunk head believed to be stable.

Upvotes: 2

Related Questions