CODEWITHSUNDEEP
airflowgoogle-cloud-composer
Montenegrodr
Montenegrodr

Reputation: 1636

Composer Scheduler is not triggering my DAGs within the planned time

I have the following configuration and the DAG isn't running in the expected time:

import pendulum


default_args = {
    "start_date": pendulum.today(),
     "retries": 0,
}

schedule_interval: str = "1 0 * * *"

It has been two days the Next Run: 2022-08-25, 00:01:00 shows the correct day but it is not triggered.

Version: 2.2.5+composer

Upvotes: 0

Views: 368

Answers (2)

Hussein Awala
Hussein Awala

Reputation: 5110

As explained in the other answer, the problem is with the start_date which is always > run_date, if you don't want to fix the date for some reasons, you can just replace pendulum.today() by pendulum.yesterday().

Upvotes: 0

Elad Kalif
Elad Kalif

Reputation: 16119

Do not use dynamic start_date. This is explained in depth in Airflow FAQ.

Specifically for your case since you are after a daily run, If you want the first DAGRun to start on 2022-08-05 at 01:00 AM then you should set start_date as 1 day behind since Airflow trigger runs at the end of the interval:

from datetime import datetime

default_args = {
    "start_date": datetime(2022, 08, 04, 01),
     "retries": 0,
}

dag = DAG(
    ...,
    schedule_interval="1 0 * * *",
    default_args=default_args,

)

Note that you can also do:

from datetime import datetime

default_args = {
     "retries": 0,
}

dag = DAG(
    ...,
    schedule_interval="1 0 * * *",
    start_date=datetime(2022, 08, 04, 01),

)

Upvotes: 3

Related Questions