CODEWITHSUNDEEP
python-3.xairflowairflow-scheduler
Maile Cupo
Maile Cupo

Reputation: 748

Airflow DAGS not executing according to schedule

Some of my airflow dags are not being executed per the schedule:

enter image description here

As you can see, there are two dags here that should have run at 5:00 and 8:00 utc on the 02-11 but did not. Here is the relevant dag definition info for the one scheduled for 0 5,17 ***:

default_args = {
    'owner' : 'airflow',
    'retries' : 1,
    'retry_delay' :timedelta(minutes=5)
}

dag = DAG(
    dag_id = '*********',
    start_date = airflow.utils.dates.days_ago(2),
    default_args = default_args,
    schedule_interval = '0 5,17 * * *'
)

As you can see, the 5:00 run on 2-11 is more than a full schedule_interval past the start_date, so there is no reason it should not have triggered per this astronomer article and all the other documentation out there.

Upvotes: 1

Views: 631

Answers (1)

MJK
MJK

Reputation: 1401

The airflow schedule interval could be a challenging concept. Basically airflow won't trigger the DAG at the starting time of your schedule period instead it will trigger at end of your schedule period.

For example if you schedule a daily batch job at 2 o'clock it will trigger your DAG at the end of the schedule period which means it will trigger 24th hours (next day 2 o'clock), Same if it hourly job then it will trigger after 60 minutes completed, so your DAG will be triggered 60th minute which means next hour.

Same way in your example, you scheduled 5th and 17th hour so it will trigger once the schedule period is completed, in this case 5th hour run going to trigger at 17th hour and 17th hour run going to trigger at next day 5th hour.

Upvotes: 1

Related Questions