CODEWITHSUNDEEP
pythonairflow
nven
nven

Reputation: 1227

Using an Airflow sensor to start a DAG run

I'm trying to poll an HTML API endpoint for new data, and only execute the DAG if new data is present.

I have a sensor with a standard poke function that will return True if this is the case.

I'm wondering if it is possible to avoid having the task scheduled, and only execute the task when the sensor returns true? Currently, I'm running the DAG daily and have the sensor timeout after 24h (so there is only ever one concurrently running DAG). However, in case new data comes in twice per day, it will have to wait until the next DAG run to be processed.

Upvotes: 2

Views: 2257

Answers (1)

Ryan Yuan
Ryan Yuan

Reputation: 2566

You can use 1 DAG dedicated to sensing. And use another dag to do processing.

Sensing DAG:

sensor keeps poking -> once poke() returns true, use TriggerDagRunOperator to trigger Sensing DAG -> use TriggerDagRunOperator to trigger Processing DAG

Processing DAG:

process whatever you want

Upvotes: 4

Related Questions