CODEWITHSUNDEEP
airflowairflow-scheduler
Aviv Oron
Aviv Oron

Reputation: 565

unwanted DAG runs in Airflow

I configured my DAG like this:

default_args = {
    'owner': 'Aviv',
    'depends_on_past': False,
    'start_date': datetime(2017, 1, 1),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=1)
}

dag = DAG(
    'MyDAG'
    , schedule_interval=timedelta(minutes=3)
    , default_args=default_args
    , catchup=False
)

and for some reason, when i un-pause the DAG, its being executed twice immediatly. Any idea why? And is there any rule i can apply to tell this DAG to never run more than once in the same time?

Upvotes: 2

Views: 1380

Answers (2)

Duwa
Duwa

Reputation: 76

I think its because you have missed the scheduled time and airflow is backfilling it automatically when you ON it again. You can disable this by catchup_by_default = False in the airflow.cfg.

Upvotes: 0

x97Core
x97Core

Reputation: 1514

You can specify max_active_runs like this:

dag = airflow.DAG(
    'customer_staging',
    schedule_interval="@daily",
    dagrun_timeout=timedelta(minutes=60),
    template_searchpath=tmpl_search_path,
    default_args=args,
    max_active_runs=1)

I've never seen it happening, are you sure that those runs are not backfills, see: https://stackoverflow.com/a/47953439/9132848

Upvotes: 2

Related Questions