CODEWITHSUNDEEP
airflow
Shreyas Kaushik
Shreyas Kaushik

Reputation: 15

Airflow parallelism based on operator type

Does airflow support throttling or parallelism limit by operator type. I want to limit number of spark submits across different dags but not end up limiting the parallelism across

Upvotes: 1

Views: 52

Answers (1)

n8sty
n8sty

Reputation: 1438

You want to use an Airflow pool. See for details: https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#pools.

In short, what a pool does is limit the number of available slots for tasks to run in. In your case, you can subclass the operators in question and give them a default pool parameter of a particular pool instance so that developers are nudged in the correct direction.

Upvotes: 1

Related Questions