Reputation: 1227
I'm trying to poll an HTML API endpoint for new data, and only execute the DAG if new data is present.
I have a sensor with a standard poke function that will return True
if this is the case.
I'm wondering if it is possible to avoid having the task scheduled, and only execute the task when the sensor returns true? Currently, I'm running the DAG daily and have the sensor timeout after 24h (so there is only ever one concurrently running DAG). However, in case new data comes in twice per day, it will have to wait until the next DAG run to be processed.
Upvotes: 2
Views: 2252
Reputation: 2566
You can use 1 DAG dedicated to sensing. And use another dag to do processing.
Sensing DAG:
sensor keeps poking -> once poke() returns true, use TriggerDagRunOperator to trigger Sensing DAG -> use TriggerDagRunOperator to trigger Processing DAG
Processing DAG:
process whatever you want
Upvotes: 4