CODEWITHSUNDEEP
pythonairflow
Tevett Goad
Tevett Goad

Reputation: 156

Is it possible to kill the previous DAG run if its still running when its time for the latest run?

Our airflow is forced to interact with a company with a very poor system. Its not unusual for our DAG to get stuck waiting for a report that never actually gets completed. This DAG runs daily pulling the the same information, so if its time for the next run it would be nice to just kill the last run and move on with the new one. I haven't found anything saying Airflow has a DAG argument that can achieve this. Is there a quick easy setting for this behavior, or would it need to be done logically in the sensor that checks if the report is complete?

Upvotes: 1

Views: 1088

Answers (1)

qfwfq
qfwfq

Reputation: 2516

If your DAG is scheduled daily, how about setting dagrun_timeout to 24 hours? I believe this should in effect kill the previous dag run around when it kicks off a new one. Related question about setting DAG timeouts.

Alternatively, you could either use a PythonOperator, define your own operator, or extend the report sensor you describe to kill the previous DagRun programmatically. I believe that this would look like...

  1. Get the current dag run from the Airflow context
  2. Get the previous dag run with dag_run.get_previous_dagrun()
  3. Set the state on the previous dag run with prev_dag_run.set_state

My recommendation would be to set the timeout given these two options. I agree that there is no specific kill_previous_run dag argument

Upvotes: 1

Related Questions