CODEWITHSUNDEEP
pythonairflow
abkf12
abkf12

Reputation: 221

How to have multiple branches in airflow?

I'm trying to figure out how to manage my dag in Apache Airflow.

There are a few master steps that I need to. for example, let's say step 1 and step 2 should always be executed before branching out. And then steps A, B, C, D, E should all be concurrently branched out after step 2, and each of these have their own steps.. so like A1, A2.. same with B~E.

What's the best way of managing it? It looks like there is BranchOperator but it says it's really good for alerts, and I'm not trying to do alerts here but actually use operators for A1, A2.. steps. Should A-E be all subdags? If so, how do I trigger subdags?

Upvotes: 2

Views: 5269

Answers (2)

floating_hammer
floating_hammer

Reputation: 439

Do you want to concurrently execute all the Tasks A,B,C..etc? If so, then branchOperator may not be suitable. branchOperator selects a particular branch based some logic.

Upvotes: 1

amoskaliov
amoskaliov

Reputation: 799

If you want structure like this:

               -> A1 -> A2 -> ...
              /
step1 -> step2 -> B1 -> B2 -> ...
              \ 
               -> C1 -> C2 -> ...

Then you can just set dependency in your DAG as

step1 >> step2
step2 >> A1 >> A2 >> ...
step2 >> B1 >> B2 >> ...
step2 >> C1 >> C2 >> ...

BranchOperator is used when you have conditional branches (if X then execute A else B).

Upvotes: 3

Related Questions