Django background tasks with Twisted

Question

I'm making a web app using Django.

I'd like events to trigger 'background' tasks that run parallel to the Django application. (by parallel I just mean that they don't impact the speed of the users experience)

Types of tasks I'm talking about

• a user logs in and an event is trigged to start populating that users cache in anticipation of future requests based on their use habits.

• a user posts some data to the database but that post triggers an api call to another website where the returned data will be parsed, aggregated and used to supplement that users post.

• rolling updates of data used in the app through api calls to other websites

• aggregating data and running general maintenance tasks.

After a few days of research I'm thinking that I should use twisted to accomplish this. Which has lead me to my question:

• Is twisted overkill for what I'm trying to accomplish?

Many of these tasks are far more i/o bound than cpu bound. So I'm thinking asynchronous is best.

Anyone advice would be appreciated.

Thank you

Answer 1

Yes, I think it's overkill.

Rather than fold in a full async framework such as Twisted, with all the technical overhead that brings, you might be better off using a task queue to be able to do what you want as a background process.

When your app needs to do a background task (anything that would otherwise block the request/response cycle), put the task in the queue and let a separate worker process pick things off the queue and deal with them as fast as it can. (You can always add more workers).

Two of the most popular queue libraries for Python/Django are celery and rq. They're especially good with Redis as a backend, but there are other backend options, too.

Personally, I much prefer rq over celery, in terms of its API and its clean setup, but both are used by a lot of people.

And both are definitely easier to get your head around than something like Twisted, IMO.