CODEWITHSUNDEEP
pythonetlairflow
Qin Yi
Qin Yi

Reputation: 15

How to trigger task in airflow when pre-task failed and depends_on_past = true?

I have a task needs to be triggered every 5 minutes. And the next task needs to wait for the previous one to be done (no matter success or failure on the previous task).

Here's how I configure the parameters.

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2019, 7, 1, 8, 30),
    'max_active_runs': 1,
    'depends_on_past': True,
    'execution_timeout': timedelta(seconds=300)
}

dag = DAG(
    dag_id='dag1', default_args=default_args,
    schedule_interval='*/5 8-16 * * *',
    dagrun_timeout=timedelta(minutes=600))

def task1(ds, **kwargs):
    #do something

task1 = PythonOperator(
    task_id='task1',
    provide_context=True,
    python_callable=task1,
    trigger_rule= 'all_done',
    dag=dag)

Under my configuration, task1 will be triggered while the previous state is a success, and be blocked while the previous state is failed. I found the description in airflow's doc, "trigger_rule can be used in conjunction with depends_on_past (boolean) that, when set to True, keeps a task from getting triggered if the previous schedule for the task hasn’t succeeded".

So how can I achieve the purpose of triggering task1 while the previous task1 is done and no matter what the pre-state is?

Upvotes: 0

Views: 724

Answers (1)

Jim Baldwin
Jim Baldwin

Reputation: 71

max_active_runs: 1 will only allow one execution of the dag at once, covering your waiting for previous requirement.

You have 2 depends_on_past settings. Remove one and have it set to false.

Upvotes: 1

Related Questions