CODEWITHSUNDEEP
taskairflowdirected-acyclic-graphs
William
William

Reputation: 589

How can I make sure my airflow DAG runs one day at a time?

I want to set my DAG to run one day at a time. How can I achieve this?

I tried "depends on past=True", but it only makes sure each task is run subsequently. What I want is that, if I'm backfilling from day X, all tasks of day X are run before the DAG for day X+1 can start and so on.

Upvotes: 6

Views: 2615

Answers (2)

Chengzhi
Chengzhi

Reputation: 2591

Check out this page.
You can set up your schedule_interval with the following: 0 0 * * *.
If you have set up catchup_by_default = True with start_date several days ago, airflow will backfill all your previous tasks first then run the schedule based on schedule_interval.

Upvotes: -1

x97Core
x97Core

Reputation: 1514

You can use max_active_runs to control the number of active dag runs. Limiting it to one should satisfy your use case.

dag = airflow.DAG(
    'customer_staging',
    schedule_interval="@daily",
    dagrun_timeout=timedelta(minutes=60),
    template_searchpath=tmpl_search_path,
    default_args=args,
    max_active_runs=1)

Upvotes: 9

Related Questions