CODEWITHSUNDEEP
airflowairflow-scheduler
bsd
bsd

Reputation: 1247

event listener in airflow

I am having a task which will listen for certain events and kick-off other functions.

This function (the listener) subscribes to a kafka topic and runs forever, or at least until it will get a 'stop' event.

Wrapping this as an airflow operator doesn't seem to work properly. Meaning, if I send the stop event, it will not process it, or anything else for that matter. Is it possible to run busy loop functions in airflow ?

Upvotes: 1

Views: 5080

Answers (1)

Ash Berlin-Taylor
Ash Berlin-Taylor

Reputation: 4048

No, do not run infinite loops in an Airflow task.

Airflow is designed as a batch processor - long/infinite running tasks are counter to it's entire scheduling and processing model and while it might "work", it will lock up a task runner slot.

Upvotes: 4

Related Questions