CODEWITHSUNDEEP
airflow
Forest.Lin
Forest.Lin

Reputation: 123

can trigger an airflow job in sub seconds?

I tried to trigger an airflow job more than two times in a second via airflow rest api and found that only a trigger was successful. And if scheduling an airflow job, the minimum time is 1 second.

So any approach to trigger an airflow job in sub seconds?

Upvotes: 1

Views: 348

Answers (1)

Ofek Hod
Ofek Hod

Reputation: 4024

You didn't describe your use case, but mainly saying- Apache Airflow is built for scheduled batch jobs, I suggest you don't go below the hourly interval.
If you find yourself with a need for more frequent processing, maybe Airflow is not the platform for you, and if we're talking about sub seconds, consider to choose a streaming platform, like Kafka Streams, Apache Spark, NiFi or StreamSets.

Upvotes: 1

Related Questions