CODEWITHSUNDEEP
pythoncelery
demon.mhm
demon.mhm

Reputation: 469

reuse results for celery tasks

Is there any common solution to store and reuse celery task results without executing tasks again? I have many http fetch tasks in my metasearch project and wish to reduce number of useless http requests (they can take long time and return same results) by store results of first one and fire it back without real fetching. Also it will be very useful to does not start new fetch task when the same one is already in progress. Instead of running new job app has to return AsyncResult by id (id is unique and generated by task call args) of already pending task.

Looks like I need to define new apply_async(Celery.send_task) behavior for tasks with same task_id:

  1. if task with given task_id doesn't started yet then start it
  2. if task with given task_id already started return AsyncResult(task_id) without actually run task
  3. @task decorator should accept new ttl kwarg to determine cache time (only for redis backend?)

Upvotes: 3

Views: 1828

Answers (1)

Benjamin Cbr
Benjamin Cbr

Reputation: 536

Looks like the simplest answer is to store your results in a cache (like a database) and first ask for the result from your cache else fire the http request.

I don't think there's something specific to celery that can perform this.

Edit:

To comply with the fact that you the tasks are sent at the same time an additional thing would be to build a lock for celery task (see Celery Task Lock receipt).

In your case you want to give the lock a name containing the task name and the url name. And you can use whatever system you want for cache if visible by all your workers (Redis in your case?)

Upvotes: 3

Related Questions