CODEWITHSUNDEEP
pythondjangobackup
Przemysław Pietrzkiewicz
Przemysław Pietrzkiewicz

Reputation: 2767

Django - consecutive dumpdata calls fail, even though work correctly when run separately

I've run into an issue trying to set up automatic backups on my site. The problem boils down to the following.

I open the Python shell and I call the dumpdata command twice. It works for the first time and it returns empty list the second time. After that all further dumpdata calls return empty list:

>>> python manage.py shell
>>> from django.core.management import call_command
>>> call_command("dumpdata")
[{"pk": 1, (...) // lots of data //
>>> call_command("dumpdata")
>>> []

To make it work again I need to restart the python shell.

Edit: I use Django 1.4 and Python 2.6

Edit2: My current hypothesis is that the problem is related to this issue: https://code.djangoproject.com/ticket/5423 - identified 5 years ago and according to Django 1.5 release notes, to-be resolved in the next release. Does anybody have an idea how to workaround this issue without altering the 1.4 framework code being run on the machine?

Edit3: However sql dump of the whole database is only 0.5 MB, which makes it rather unprobable that the serialization is running out of memory. And anyway, wouldn't I get an explicit error in such case?

Edit4: Mystery solved. As Tomasz Gandor correctly determined, the problem was that shell executes the commands in one transaction, and after one of the commands causes DBError, further DB calls are ignored, as described here: https://code.djangoproject.com/ticket/10813 . Why the DB error during the first dumpdata wasn't explicitely reported remains a mystery to me.

Upvotes: 2

Views: 546

Answers (1)

Tomasz Gandor
Tomasz Gandor

Reputation: 8823

I see that django is messing something with transactions.

I executed a simple example under the debugger:

# test.py
from django.core.management import call_command
call_command("dumpdata")
print "\n---"
call_command("dumpdata")
print

And called it like:

DJANGO_SETTINGS_MODULE=settings python test.py > log.txt

My log.txt ended with "---\n[]\n"

After running it in the debugger i found, deep down in django.core.management.commands.dumpdata.handle() that model.objects.all() keeps returning [].

I called model.objects.iterator(), and got the exception:

(Pdb) list(model.objects.iterator())
*** Error in argument: '(model.objects.iterator())'
(Pdb) p list(model.objects.iterator())
*** DatabaseError: DatabaseError('current transaction is aborted, commands ignored    until end of transaction block\n',)
(Pdb)

So, I hacked together code, which plays with the transaction itself:

# test.py version 2.0!

#!/usr/bin/env python
# from django.core.management import call_command
import django.core.management as mgmt
from django.db import transaction

''' 
try:
    import settings # Assumed to be in the same directory.
except ImportError:
    import sys
    sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n(If the file settings.py does indeed exist, it's causing an ImportError somehow.)\n" % __file__)
    sys.exit(1)
'''

@transaction.commit_manually
def main():
    # mgmt.call_command('dumpdata', use_base_manager=True)
    mgmt.call_command('dumpdata')
    transaction.rollback()
    print
    print '---'
    print
    """ 
    mgmt._commands = None
    import sys
    reload(sys.modules['django.core.management.commands.dumpdata'])
    """
    mgmt.call_command('dumpdata')
    transaction.rollback()
    print

if __name__ == "__main__":
    main()

This spits out the whole database - every time!

Upvotes: 1

Related Questions