CODEWITHSUNDEEP
pythonmysqldjangomysql-pythondjango-orm
Michael Waterfall
Michael Waterfall

Reputation: 20569

Django DB transactions and deadlocks

When running Django with Gunicorn with multiple processes/workers I'm getting a deadlock issue with some of my manual MySQL database transactions.

DatabaseError(1205, 'Lock wait timeout exceeded; try restarting transaction')

My setup uses multiple databases, and my function needs to be passed the database to use when it's called. For this reason, I can't use the standard Django transaction decorators as the db needs to be hard-coded as an argument. I've inspected the decorator code to look at how transactions are managed, and my function looks like this:

from django.db import connections

def process(self, db, data):

    # Takeover transaction management
    connections[db].enter_transaction_management(True)
    connections[db].managed(True)

    # Process
    try:
        # do things with my_objects...
        for obj in my_objects:
            obj.save(using=db)
        connections[db].commit()
    except Exception as e:
        connections[db].rollback()
    finally:
        connections[db].leave_transaction_management()

Can anyone spot what may be going wrong here?

Upvotes: 1

Views: 3092

Answers (1)

harmv
harmv

Reputation: 1922

Note you might want to use the more clear with-style syntax. The following should be identical to your code above, but more pytonic.

from django.db import transaction
from __future__ import with_statement

def process(self, db, data):

    with transaction.commit_on_success(using=db):
        # do things with my_objects...
        for obj in my_objects:
            obj.save(using=db)

or with a decorator

from django.db import transaction

@transaction.commit_on_success(using=db)
def process(self, db, data):    

    # do things with my_objects...
    for obj in my_objects:
        obj.save(using=db)

That does not solve your deadlock problem though..

You might have success with lowering your transaction isolation level. This defaults on mysql to REPEATABLE READ wich is too strict for most usages. (oracle defaults to READ COMMITTED')

You can achieve this by adding this to your settings.py

MYSQL_DATABASE_OPTIONS = {'init_command': 'SET storage_engine=INNODB; SET 
                 SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;'}

DATABASES = {
  'default': {  # repeat for each db
       'ENGINE':  ... etc
       ...
       ...
       'OPTIONS': MYSQL_DATABASE_OPTIONS
      }
  }

Upvotes: 4

Related Questions