CODEWITHSUNDEEP
pythondjangotaskcelerybackground-task
Bolshoi Booze
Bolshoi Booze

Reputation: 514

Django Background tasks vs Celery

I am trying to do some tasks in django that consume alot of time. For that, I will be running background tasks.

After some R&D, i have found two solutions:

Both options seem to fulfill the criteria but setting up Celery will require some work. Now as far as the second option is concerned, setup is fairly simple and in fairly quick amount of time, i can go on writing background tasks. Now my questions if i adopt the 2nd option is this:

  1. How well does Django Background tasks perform ? (Scalability wise in Production environment).
  2. Can i poll the tasks (after some time) in DB to check the task's status?
  3. Architecture of Django-Background-tasks? Couldn't find any clear explanation about it's architecture (Or have I missed some resource?)
  4. Again coming to the first point, how well does Django Background tasks perform in production. (Talking about prior experience of using this in prod.)

Upvotes: 10

Views: 7469

Answers (3)

Kagan Coskun
Kagan Coskun

Reputation: 451

I have decided to use Django-Background-Tasks. Let me clarify my motivations.

The tasks that will be processed by Django-Background-Tasks doesn't need to be processed in a fast manner. As it is stated by the name, they are background tasks. I accept delays.

The architecture of Django-Background-Tasks is very simple. When you call a method to be process in the background in your code a task record is inserted to the Django-Background-Tasks tables in your database. And the method you called is not executed actually. It is proxied. Then you should trigger another process to execute the jobs. Your method is then executed in this process.

The process that execute jobs can be executed by a cron entry in your server.

Since this setup is so easy and work for I decided to use Django-Background-Tasks. But If I needed something more responsive and fast I would use Celery since it is using memory and there is an active process that processes the jobs. Which isn't the case in Django-Background-Tasks.

Upvotes: 3

Umair Mohammad
Umair Mohammad

Reputation: 4635

  1. How well does Django Background tasks perform ? - This will depend upon how and what you implement. One thing to note is, Django-background-tasks is based upon database where celery can have redis/rabbitmq as backend, so most probably we'll see considerable performance difference here.

  2. Can I poll the tasks (after some time) in DB to check the task's status? - It's possible in celery and maybe you can find a solution by inspecting django-background-tasks internal code. But one thing is, we can abort celery task, which maybe not possible in Django-Background-tasks.

  3. Architecture of Django-Background-tasks? Couldn't find any clear explanation about it's architecture (Or have I missed some resource?) - It's simple Django based project. You can have a look at code. It's seems to be pretty straightforward.

  4. Again coming to the first point, how well does Django Background tasks perform in production. - Haven't used in production. But since Django-Background-tasks is database based and celery can be configured to use redis/rabbitmq - I think celery have a plus point here.

To me this comparison, seems to be link comparing pistol with a high-end automatic machine guns. Both do same job. But one simple straightforward - other little complicated but with lots of options and scope.

Choose based on your use case.

Upvotes: 7

Nino Walker
Nino Walker

Reputation: 2903

Setting up celery takes work (although less when using Redis). It's also serious tool with almost a decade of investment and widespread industry adoption.

As for performance, scaling behaviors of task systems which are backed by queues vs those backed by RDBMs are well understood – but may not be relevant to you as "scalability" is a very subjective term. This thread provides some good framing on the subject and questions.

Comparing stars on GitHub (bg tasks' 3XX vs Celery's 13XXX), you should realize Django-Background-tasks has a smaller user base, and you're probably going to need to get into the internals to understand the architecture and precise mechanics. That shouldn't stop you – just be prepared to DIY when answers aren't forthcoming.

Upvotes: 9

Related Questions