CODEWITHSUNDEEP
airflowairflow-scheduler
Mathias Andersen
Mathias Andersen

Reputation: 526

Airflow scheduler wont queue some tasks that have been cleared in the UI

I have a bunch of DAG runs in the ui. If I clear some task across all the DAG runs, some of the tasks are correctly triggered, whereas others are stuck with a cleared state.

At the moment I am simply using airflow CLI to backfill these tasks. This works, but it is unfortunate that I need an unbroken CLI session to complete a clearing/reprocessing scenario.

Upvotes: 3

Views: 573

Answers (1)

Mathias Andersen
Mathias Andersen

Reputation: 526

The reason for this is the naming (and thus type) of your DAG runs.

If you go into your airflow meta data db and open table "dag_runs", you will see run_id. The scheduler identifies the runs it creates with "scheduled__" followed by a datetime. If you backfill the run_id will be named "backfil_" followed by a datetime.

The scheduler will only check and queue tasks for run_ids that starts with "scheduled__", denoting a DagRunType of "scheduled".

If you rename the run_id from backfill_ to scheduled__, the scheduler will identify the dag runs and schedule the cleared task underneath.

This SQL query will change bacfill_ to schelduled__:

UPDATE dag_run
SET run_id = Replace(run_id, 'backfill_', 'scheduled__')
where id in (
    select id from dag_run where ("run_id"::TEXT LIKE '%backfill_%')); 
-- note that backfill_ is a single underscore, and scheduled__ is two. 
-- This is not a mistake in my case. But please review the values in your tabel.

Upvotes: 1

Related Questions