CODEWITHSUNDEEP
airflow-schedulerairflow
Jitesh Sharma
Jitesh Sharma

Reputation: 75

Branches created using BranchPythonOperator do not merge?

I am using BranchPythonOperator to make branches in airflow. My use case is I need to make two branches from mainstream. Branch A (which has few tasks) will be followed when somefile.csv is present otherwise Branch B(which has no task) to follow. At last both branches should be merged to make a mainstream again. Now I am able to follow either Branch A or Branch B but the issue is if I follow Branch B final mainstream tasks executed and if I follow Branch A final mainstream tasks are being skipped.

MainstreamTaskA.setDownStream(MainstreamTaskB)
MainstreamTaskB.setDownStream(BranchATaskA)
BranchATaskA.setDownStream(MainstreamTaskC)
MainstreamTaskB.setDownStream(MainstreamTaskC)

I have set trigger rule as "all_done" in MainstreamTaskB and MainstreamTaskC.

Can somebody guide me through this?

Upvotes: 0

Views: 738

Answers (1)

kaxil
kaxil

Reputation: 18824

I cannot see the other branch in your dependencies. The only branch is BranchATaskA. But based on what you have mentioned you should have the following task dependencies and have two branch tasks BranchATaskA and BranchATaskB.

MainstreamTaskA >> MainstreamTaskB
MainstreamTaskB >> BranchATaskA >> MainstreamTaskC
MainstreamTaskB >> BranchATaskB >> MainstreamTaskC

You should have trigger rule as all_done on MainstreamTaskC.

Upvotes: 1

Related Questions