CODEWITHSUNDEEP
airflowairflow-scheduler
Yug
Yug

Reputation: 135

simultaneous run of same DAG is possible in airflow?

We have one airflow DAG which is accepting input from user and performing some task. we want to run same DAG simultaneous with different input from user. we found multiple links for simultaneous task run but not able to get info about simultaneous run. so if we triggered DAG with two diff inputs from cli then its running fine with two instances but just want to understand that both instances are running independently or its waiting for one instance to finish and then trigger the other ?

Upvotes: 8

Views: 8966

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 16139

All of what you mention can be done. Tasks can be executed in parallel.

It's just the proper configuration of max_active_runs wait_for_downstream and depends_on_past

DAG parameters:

max_active_runs - maximum number of active DAG runs

if you just want the DAGs to be able to execute two jobs in parallel (with no conditions between two distinct runs) then set max_active_runs=2

Operators parameters:

wait_for_downstream - when set to true, an instance of task X will wait for tasks immediately downstream of the previous instance of task X to finish successfully or be skipped before it runs

depends_on_past - when set to true, task instances will run sequentially and only if the previous instance has succeeded or has been skipped.

Upvotes: 10

Related Questions