CODEWITHSUNDEEP
javaapache-flinkflink-streaming
flinkexplorer
flinkexplorer

Reputation: 95

how to schedule a flink batch processing periodically

I have a java application which does a flink batch processing of a batch obtained by querying the tables from the database and feed it into a kafka topic. How would I make this scheduled periodically. Is there a flink scheduler? For example, my java application should keep running in the background and the flink scheduler should periodically query the tables from the database and flink batch process it and feed into kafka(flink batch processing and feeding into kafka is already done part of my application). Please help if anyone has pointers on this.

Upvotes: 4

Views: 3346

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18987

Flink does not provide a job scheduler.

Have you considered implementing the use case with a continuously running Flink DataStream application? You could implement a SourceFunction that periodically queries the database.

Continuous streaming applications have the benefits of fewer moving parts (no scheduler, no failure handling if something goes wrong) and a consistent view across the boundaries of "batches". The down side is that the job is always consuming resources (Flink is not able to automatically scale-down at low load).

Upvotes: 5

Related Questions