CODEWITHSUNDEEP
airflowairflow-schedulerdirected-acyclic-graphs
JaMo
JaMo

Reputation: 105

How to run non-dependent sibling tasks sequentially?

Let's suppose I have a dag with 4 tasks, and the structure would be more or less like this if we only care about the order in which they are run:

t1.set_downstream(t2)
t2.set_downstream(t3)
t3.set_downstream(t4)

This let's me run the tasks sequentially t1 -> t2 -> t3 -> t4

My issue is, I want them to run in the above sequence, but I want t4 to ignore if and only if t3 fails.

I.e: the graph would look like this if we only care about dependencies

t1.set_downstream(t2)
t2.set_downstream(t3)
t2.set_downstream(t4)

What can I do to have both scenarios? My current option is to set the trigger rule of t4 to all_done, however, I don't want it to run if t1 or t2 fails.

Any ideas? Is this even possible?

Upvotes: 0

Views: 193

Answers (1)

aneto
aneto

Reputation: 26

You can use a BranchOperator to add the logic you are looking for. Or even create your own SkipOperator based on SkipMixin.

Upvotes: 1

Related Questions