CODEWITHSUNDEEP
pythondjangounit-testingceleryintegration-testing
Michael Platt
Michael Platt

Reputation: 1342

Django unit test query against production database

I am trying to write some unit tests for an asynchronous method in my Django app. Essentially, when a user does a certain POST, this method is kicked off so the application doesn't hang. What I want to test is the ability to cancel this request as it is running (in this case via celery). The only problem with doing this is because Celery is running independent of the web application, it pushes the results to the real database, not the test database that is created by Django Unit Tests. So what I'm wondering is how would I be able to do something like 

Results.objects.get(id=some_id)

and tell it to point to the actual database. I've already tried the following:

Results.objects.using('default').get(id=some_id)

Thinking that maybe this would solve the problem, but it didn't find the result (again because it was pushing the information to the actual database, not the test DB). I did some searching around and found this link: How do I run a unit test against the production database? but the answers on here say things like "You shouldn't test against production." I completely understand this is not good practice but I need to hit the live database. Everything is being run on a VM so it's all a test database anyways. Does anyone know of a way to hit the real database?

Thanks,

Upvotes: 1

Views: 1429

Answers (3)

Anshul Warade
Anshul Warade

Reputation: 11

There is one thing we can do, a dirty trick!

  • Run your celery worker and scheduler on a setting which uses say 'new_one' database.

  • You should have a settings.py like this -

    DATABASES = {
   'default': {
       'TEST': {
           'NAME': 'new_one',
       },
   } 
} 
db_from_env = dj_database_url.config(default=config('DATABASE_URL'))
DATABASES['default'].update(db_from_env)

  • After that run your django server on another database say
    'another_new_one'.

  • Run your test cases, they will pass. Why? Test cases are using 'new_one' database, celery also using 'new_one' database, your django server using 'another_new_one'. Cheers!

  • I have tried giving same names to test and production database, but error comes when running tests 'database already in use by someone else', that someone else is django server. So need two different databases, 1- new_one 2-another_new_one

      Note- I have used .env file to use config(), where I have my production database name host user password etc. It can change according to what you use. Just use two different databases and do this dirty trick, it will work. As it worked for me.

Upvotes: 1

Sovetnikov
Sovetnikov

Reputation: 448

If you really want to run dedicated celery worker process from unit test, then you need to start celery worker from setUp() (or whatever you want method) and copy django.conf.settings.DATABASE from test process to celery worker process.

For starting point see my answer on very similar question https://stackoverflow.com/a/42107423/590233 (there is code that starts celery worker)

And here is sample code that configures django and DATABASE settings within single module https://gist.github.com/Sovetnikov/369a8d05ba2b6482fa20769bc498f122

Just combine two techniques and you will get what you want.

Upvotes: 0

dm03514
dm03514

Reputation: 55962

I'm not sure if testing against your prod is a better solution than running your celery workers in dev mode against your test database. How are you starting your celery workers ?? I imagine you're starting your celery workers external of the test using your prod/real settings, then you're running your tests using the test settings. For an integration test, it would probably be more repeatable/maintainable if you had an integration environment.

a common test strategy for this would be:

  • unit test your view to make sure that it calls the correct celery task with the correct arguments, you could stub out the celery task dependency, or you could run the celery task syncronously using CELERY_ALWAYS_EAGER=True, and an in memory Queue.
  • if you stub out the celery dependency, then there should be a separate unit test exercising its logic

Your original scenario is not super well suited for django unit test framework because of the inter process dependency of the celery broker. An integration test between your app and celery would probably be valuable, but for it to be maintainable, repeatable, and reliable celery broker process startup, cleanup, etc should probably be managed in an automated way.

Upvotes: 0

Related Questions