Structuring Airflow DAG with complex trigger rules

Question

This is probably as much a logic problem as anything, but it has me stumped. I have the following dag:

I have two main branching events that are tripping me up:

• Following A, only the branch B or C should run.
• Following B, D should optionally run before E.

Right now I have this implemented so that E has the trigger rule "none_failed", which was needed to prevent E from being skipped when D is skipped. This set up works for all cases except when C needs to run. In this case, C and E run simultaneously (you can see via the box colors B/D are skipped as intended, but E is green). After reading the trigger rule docs, I understand why this is happening (both E's parent tasks are skipped, therefore it runs). However I can't seem to figure out how to get the intended behavior out of this (e.g. keep the current behavior of B/D/E when that branch is run but don't run E when C is run).

For additional context, this is not my entire DAG. Tasks C and E converge into another task downstream of these with a trigger rule of ONE_FAILED, but I omitted this from the example for simplicity. Any ideas how to get the intended behavior?

Answer 1

This is probably not the best solution but seems it covers all of your scenarios. Main thing is that I added a dummy task before E in order to control E's timing and change trigger_rule for E to be "one_success".

"one_success" requires at least 1 immediate parent to be succeeded, so for E, either D or dummy has to success in order for E to run.

A = BranchPythonOperator(task_id='A', python_callable=_branch_A, dag=dag)

B = BranchPythonOperator(task_id='B', python_callable=_branch_B, dag=dag)
C = DummyOperator(task_id='C', dag=dag)

D = PythonOperator(task_id='D', python_callable=_test, dag=dag)
dummy = DummyOperator(task_id='dummy', dag=dag)
E = DummyOperator(task_id='E', trigger_rule='one_success', dag=dag)

A >> [B, C]
B >> [D, dummy] >> E

Demo