CODEWITHSUNDEEP
pythoncelery
Alex Haller
Alex Haller

Reputation: 315

Using celery to send tasks from component A to component B

The technology I would like to use in this example is Celery for queueing and python for component implementation.

Imagine a simple project hat exists of 2 components. One is a web app that connects to an API and gathers data. Component 2 is a processor that can then process the data. When the web app has gotten a piece of data from the API it is supposed to send a task into a task queue including the just crawled data which is then consumed by the processor to process the Data.

Whether or not this is a sensible way to go about a task like this is debatable and not the point of my question. My question is, the tasks to process things are defined within the processor since they state what processing function shall be executed and the definition of that function is obviously within the processor. Now that the web app doesn't have access to the task definition how does he communicate the task to the processor?

Do you have to hold a copy of the source code of the processor within the web app?

Do you make the processor a dependency of the web app?

What is the best practice approach to handle such a scenario?

Upvotes: 1

Views: 87

Answers (1)

DejanLekic
DejanLekic

Reputation: 19787

What you are describing is probably one of the most common use-cases for Celery. Just look how many people are asking Django/Flask + Celery questions here on StackOverflow... If you are a Django user, there is an entire section in the Celery documentation describing how to do exactly what you want. Things should be similar with other frameworks.

Do you have to hold a copy of the source code of the processor within the web app?

As far as I know you do not have to (I do not use any web framework) but it could be that you do need to because of some deeper integration with Celery. If your web application knows the Celery task name, and its parameters, it can schedule it to run without actually having access to the Python code. This is accomplished using send_task(task_name, ...).

Do you make the processor a dependency of the web app?

As I wrote above there are several ways to use it. If you want tighter integration then yes. If you just want to run task and get result using the send_task() than your web application should only depend on Celery.

What is the best practice approach to handle such a scenario?

Follow the Django guide. I advise you to run Celery independently, run some tasks, just so you learn about basic principles how it distributes the work, etc.

Upvotes: 1

Related Questions