CODEWITHSUNDEEP
pythonpython-3.xairflowairflow-scheduler
Gerardo Reichl
Gerardo Reichl

Reputation: 80

Clearing Tasks Airflow

I'm working with Python3 and Airflow. I've a daily process that has some individual task that fails sometimes.

I would like to know, if someone knows, how to get a list of the failed tasks, at least at the end of the process. I could change my code. I could somehow return a dict with every task name and their result, but I think it has to be an easier way.

Once the process has "finished", I want it to clear those tasks state(I mean, Failed and Upstream Failed tasks) and retry them automatically.

I've read about something about:

airflow clear -t task_name <dag_name>

So think I just have to iterate over the task names clearing them, but I need to know who they are.

Lots of thanks!

Upvotes: 3

Views: 7343

Answers (1)

joebeeson
joebeeson

Reputation: 4366

If it were me I would write my own Python script which interfaces with Airflow by loading up its models (airflow.models.TaskInstance), and database connection airflow.settings.Session, locating the failed tasks and then clearing them through the script. Airflow already has code for clearing tasks that may offer a helpful starting point.

Upvotes: 1

Related Questions