CODEWITHSUNDEEP
pythonairflowairflow-scheduler
user3735871
user3735871

Reputation: 317

Airflow - how to skip certain tasks

For a pipeline like below, sensor was set to softfail= True I'm trying to figure out how to only skip certain tasks when the sensor fails. For example only have B and D fail but still execute C and E. Many thanks for your help.

Sensor A >> B >> C >> D >> E

Upvotes: 1

Views: 650

Answers (1)

zzob
zzob

Reputation: 1134

I think you could use TriggerRule (https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-trigger-rules)
e.g

with DAG(dag_id='branch',
         start_date=datetime(2022, 8, 30),
         max_active_runs=1,
         schedule_interval=None,
         catchup=False
         ) as dag:

    #DummyOperators
    a = DummyOperator(task_id='a')
    b = DummyOperator(task_id='b') # assume failed
    c = DummyOperator(
        task_id='c',
        trigger_rule=TriggerRule.one_failed
    )
    
    a >> b >> c

Upvotes: 1

Related Questions