CODEWITHSUNDEEP
airflow
Chandra Sekhar Manginipalli G
Chandra Sekhar Manginipalli G

Reputation: 13

Airflow - pause/unpause individual dagruns of the same DAG running in parallel

We are currently evaluating airflow for a project. I was wondering if there is a way to stop/start individual dagruns while running a DAG multiple times in parallel. Pause/unpause on dag_id seems to pause/unpause all the dagruns under a dag. Instead we want to pause individual dagruns (or tasks within them). Let me know if this is achievable in airflow.

If its not possible, here are other alternatives I am thinking of, let me know your opinion on these

  1. Change task state. – Change all tasks under a dagrun to Mark Failed or Success. That way that particular dagrun is stopped on its tracks without affecting other dagruns.
  2. Airflow sensor to pull this information from s3 or http or sql or somewhere to pause current dagrun. And have a task to check on s3 everytime if this dagrun needs to be stopped (not other dagruns).
  3. subdags. - Can we pause/unpause subdags. That way for each parallel user's request we want to do we issue a subdag and we can pause userAs subdag without impacting other user’s subdags.

Upvotes: 1

Views: 1954

Answers (1)

joebeeson
joebeeson

Reputation: 4366

There's nothing "baked" into Airflow to support this but you could (ab)use the state of the DagRun by changing it to "failed" to pause and then back to "running" to resume; you won't be able to blanket unpause but for testing it should be workable.

Upvotes: 1

Related Questions