CODEWITHSUNDEEP
airflow
bsd
bsd

Reputation: 1247

Schedule a DAG in airflow to run every 5 minutes

I have a DAG in airflow and for now it is running each hour (@hourly). Is it possible to have it running each 5 minutes ?

Upvotes: 27

Views: 57707

Answers (3)

Rony Armon
Rony Armon

Reputation: 338

Airflow 2 (I'm using 2.4.2) supports timedelta for scheduling DAGs on a particular cadence (hourly, every 5 minutes, etc.) so you can put:

schedule_interval = timedelta(minutes=5)

Upvotes: 4

Jan_V
Jan_V

Reputation: 4406

The documentation states:

Each DAG may or may not have a schedule, which informs how DAG Runs are created. schedule_interval is defined as a DAG arguments, and receives preferably a cron expression as a str, or a datetime.timedelta object.

When following the provided link for CRON expressions it appears you can specify it as */5 * * * * to run it every 5 minutes.

I'm not familiar on the matter, but this is what the documentation states.

Upvotes: 3

Mike
Mike

Reputation: 2604

Yes, here's an example of a DAG that I have running every 5 min:

dag = DAG(dag_id='eth_rates',
          default_args=args,
          schedule_interval='*/5 * * * *',
          dagrun_timeout=timedelta(seconds=5))

schedule_interval accepts a CRON expression: https://en.wikipedia.org/wiki/Cron#CRON_expression

Upvotes: 43

Related Questions