How do I fix IntegrityErrors during the Django TestCase teardown?

Question

I'm porting a large webapp from Python 2.7 to Python 3.6. I have managed the database migrations and converted the code using 2to3, but I'm having problems running my testing suite. For a large number of tests, I get errors like so:

  62 self = <django.db.backends.utils.CursorWrapper object at 0x7fc7d0abbeb8>
  63 sql = 'SET CONSTRAINTS ALL IMMEDIATE', params = None
  64 
  65     def execute(self, sql, params=None):
  66         self.db.validate_no_broken_transaction()
  67         with self.db.wrap_database_errors:
  68             if params is None:
  69 >               return self.cursor.execute(sql)
  70 E               psycopg2.IntegrityError: insert or update on table "probex_historicalproject" violates forei
     gn key constraint "probex_historicalpro_history_user_id_88371c5c_fk_auth_user"
  71 E               DETAIL:  Key (history_user_id)=(303) is not present in table "auth_user".
  72 
  73 ../../venv3/lib/python3.6/site-packages/django/db/backends/utils.py:62: IntegrityError
  74 
  75 The above exception was the direct cause of the following exception:
  76 
  77 self = <django.test.testcases.TestCase testMethod=__init__>
  78 
  79     def _post_teardown(self):
  80         """Performs any post-test things. This includes:
  81 
  82             * Flushing the contents of the database, to leave a clean slate. If
  83               the class has an 'available_apps' attribute, post_migrate isn't fired.
  84             * Force-closing the connection, so the next test gets a clean cursor.
  85             """
  86         try:
  87 >           self._fixture_teardown()
  88 
  89 ../../venv3/lib/python3.6/site-packages/django/test/testcases.py:925:
  90 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  91 ../../venv3/lib/python3.6/site-packages/django/test/testcases.py:1081: in _fixture_teardown
  92     connections[db_name].check_constraints()
  93 ../../venv3/lib/python3.6/site-packages/django/db/backends/postgresql/base.py:243: in check_constraints
  94     self.cursor().execute('SET CONSTRAINTS ALL IMMEDIATE')
  95 ../../venv3/lib/python3.6/site-packages/raven/contrib/django/client.py:127: in execute
  96     return real_execute(self, sql, params)
  97 ../../venv3/lib/python3.6/site-packages/django/db/backends/utils.py:64: in execute                          
  98     return self.cursor.execute(sql, params)                                                                 
  99 ../../venv3/lib/python3.6/site-packages/django/db/utils.py:94: in __exit__                                  
 100     six.reraise(dj_exc_type, dj_exc_value, traceback)                                                       
 101 ../../venv3/lib/python3.6/site-packages/django/utils/six.py:685: in reraise
 102     raise value.with_traceback(tb)
 103 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

I have tried running the tests on some of my coworker's machines, and some of the people have no trouble running the tests, but some have the same problem as myself.

Also notable: If I run an individual test file by itself (i.e. py.test /path/to/file) then this error does not occur. I'm thinking it must have something to do with the way database data is handled between tests.

Here's an example test that is failing:

  1 
  2 from django.utils import timezone
  3 from tests.tests_probex.tests_python.testing_utilities import LoggedInTestCase
  4 from tests.tests_probex.tests_python.factories import JobFactory
  5 from probex.models import Profile, Tenant, Job
  6 from probex.views.reports import build_tenant_report
  7 from django.contrib.auth.models import User
  8 import pytest
  9 
 10 pytestmark = pytest.mark.xfail(reason="Not working in Python 3.6")
 11                                                                                                                             
 12                                                                                                                             
 13 class TestTenantReport(LoggedInTestCase):                                                                                   
 14     def setUp(self):                                                                                                        
 15         super(TestTenantReport, self).setUp()                                                                               
 16         self.end_date = timezone.now()                                                                                      
 17         self.delta = timezone.timedelta(days=7)                                                                             
 18         self.start_date = self.end_date - self.delta                                                                        
 19                                                                                                                             
 20         self.date = timezone.now() - timezone.timedelta(days=3)                                                             
 21         self.job = JobFactory(absolute_num=1600, relative_num=1600, protocol=self.protocol, project=self.project, status='Co
    mplete', created=self.date, submitter_id=self.user.id)                                                                      
 22                                                                                                                             
 23     def test_returns_new_users(self):                                                                                       
 24         users = User.objects.all()                                                                                          
 25         for user in users:                                                                                                  
 26             user.date_joined = self.date                                                                                    
 27             p = Profile.objects.get(user_id=user.id)                                                                        
 28             tenant = Tenant.objects.get(id=p.tenant_id)                                                                     
 29             tenant.app_scientist = self.user                                                                                
 30             user.save()                                                                                                     
 31             tenant.save()                                                                                                   
 32                                                                                                                             
 33         report = build_tenant_report(self.user)                                                                             
 34         self.assertGreaterEqual(report['new_users'].count(), users.count()) 

The LoggedInTestCase is a subclass of django TestCase:

 15 class LoggedInTestCase(TestCase):
 16     def setUp(self):
 17         self.user = User.objects.create_user(username='dummy@hotmail.com', email='dummy@hotmail.com', password="secret123")
 18         self.user.user_permissions.add(Permission.objects.get(codename='add_job'))
 19         self.user.user_permissions.add(Permission.objects.get(codename='change_job'))
 20         self.user.user_permissions.add(Permission.objects.get(codename='add_project'))
 21         self.user.user_permissions.add(Permission.objects.get(codename='add_user'))
 22         self.user.user_permissions.add(Permission.objects.get(codename='change_user'))                                                               
 23         self.tenant = Tenant.objects.create(name="testtenant")                                                                                       
 24         self.user.profile.tenant = self.tenant                                                                                                       
 25         self.user.profile.save()                                                                                                                     
 26         self.project = ProjectFactory(name="testproject", tenant=self.tenant, active=True, identifier='AAA')                                         
 27         self.protocol = ProtocolFactory()                                                                                                            
 28         self.root_term = TermFactory(is_root=True, acc='all')                                                                                        
 29         self.login()                                                                                                                                 
 30                                                                                                                                                      
 31     def login(self):                                                                                                                                 
 32         self.client.login(username='dummy@hotmail.com', password='secret123')                                                                        
 33                                                                                                                                                      
 34     def logout(self):                                                                                                                                
 35         self.client.logout()                                                                                                                         
 36                                                                                                                                                      
 37     def get_user(self):                                                                                                                              
 38         return self.client                                                                                                                           
 39                                                                                                                                                      
 40     def permission_restricted(self, response, first_stop_url):                                                                                       
 41         first_redirect, status_1 = response.redirect_chain[0]                                                                                        
 42         second_redirect, status_2 = response.redirect_chain[1]                                                                                       
 43         self.assertEqual(first_redirect, first_stop_url)                                                                                             
 44         self.assertEqual(status_1, 302)                                                                                                              
 45         self.assertEqual(second_redirect, reverse('home'))                                                                                           
 46         self.assertEqual(status_2, 302)   

These tests used to pass when run under Python 2.7, and I haven't changed much except for updating to 3.6. I would still expect them to pass now. If anyone has any pointers, I would greatly appreciate it. First time poster here.

Answer 1

I also encountered this issue while using the django-simple-history library. It turned out that our issue was caused by having the multiple models that depend on each other in different databases. For testing we changed the database router to set all of the models to be created in the same database, and that resolved the IntegrityError for us. I think having the history and models in different databases may also have been a problem.

Answer 2

For anyone who stumbles across this: the hint was in the keys that were violated. historicalproject and history_user are related to the django-simple-history package. We ended up updating our factory boy generators to also include a History Factory. This resolved the problem.