CODEWITHSUNDEEP
airflow
MassyB
MassyB

Reputation: 1184

how to clear failing DAGs using the CLI in airflow

I have some failing DAGs, let's say from 1st-Feb to 20th-Feb. From that date upword, all of them succeeded.

I tried to use the cli (instead of doing it twenty times with the Web UI):

airflow clear -f -t * my_dags.my_dag_id

But I have a weird error:

airflow: error: unrecognized arguments: airflow-webserver.pid airflow.cfg airflow_variables.json my_dags.my_dag_id

EDIT 1:

Like @tobi6 explained it, the * was indeed causing troubles. Knowing that, I tried this command instead: 

airflow clear -u -d -f -t ".*" my_dags.my_dag_id

but it's only returning failed task instances (-f flag). -d and -u flags don't seem to work because taskinstances downstream and upstream the failed ones are ignored (not returned).

EDIT 2:

like @tobi6 suggested, using -s and -e permits to select all DAG runs within a date range. Here is the command:

airflow clear  -s "2018-04-01 00:00:00" -e "2018-04-01 00:00:00"  my_dags.my_dag_id.

However, adding -f flag to the command above only returns failed task instances. is it possible to select all failed task instances of all failed DAG runs within a date range ?

Upvotes: 7

Views: 10752

Answers (3)

Sebastian Ziegler
Sebastian Ziegler

Reputation: 75

In airflow 2.2.4 the airflow clear command was deprecated.

You could now run:

airflow tasks clear -s <your_start_date> -e <end_date> <dag_id>

Upvotes: -1

ice
ice

Reputation: 807

One solution I've found so far is by executing sql(MySQL in my case):

update task_instance t left join dag_run d on d.dag_id = t.dag_id and d.execution_date = t.execution_date
set t.state=null,
    d.state='running'
where t.dag_id = '<your_dag_id'
  and t.execution_date > '2020-08-07 23:00:00'
  and d.state='failed';

It will clear all tasks states on failed dag_runs, as button 'clear' pressed for entire dag run in web UI.

Upvotes: 0

tobi6
tobi6

Reputation: 8239

If you are using an asterik * in the Linux bash, it will automatically expand the content of the directory.

Meaning it will replace the asterik with all files in the current working directory and then execute your command.

This will help to avoid the automatic expansion:

"airflow clear -f -t * my_dags.my_dag_id"

Upvotes: 5

Related Questions