CODEWITHSUNDEEP
airflow
Francisco Alberto
Francisco Alberto

Reputation: 37

airflow check status of previous tasks

I have a DAG with branching, so exist the case that all tasks previous to dummyjoin task has the skipped status.

The dummyjoin trigger the subsequent tasks with the all_done rule.

So I'm trying to find one rule that let me trigger the end of the DAG if all previous tasks for dummyjoin has been skipped, but I can't find it.

Or in other words, trigger the subsequent tasks only if at least one task has the success status.

The one_success from airflow trigger rule don't suit my requirement, because that rule trigger the next task AS SOON as one of the previous task has this status, and I want to wait to all previous tasks finish and then if there is one task with success status, trigger the next tasks, else end.

The tasks follow the next composition:

... >> [taskA, taskB, taskC, ..., taskN] >> dummyjoin >> subsequent_tasks

Upvotes: 2

Views: 1915

Answers (1)

Christopher Beck
Christopher Beck

Reputation: 765

What about none_failed?

From the documentation:

none_failed: all parents have not failed (failed or upstream_failed) i.e. all parents have succeeded or been skipped

If that does not work for you (if you need at least one of the tasks to have succeeded), I guess the only way is to add a second dummyjoin task in parallel to your first dummyjoin like so:

... >> [taskA, taskB, taskC, ..., taskN] >> [dummyjoin1, dummyjoin2] >> subsequent_tasks

and set the trigger_rule for dummyjoin1 to none_failed and the trigger_rule for dummyjoin2 to one_success

Upvotes: 1

Related Questions