Flink batch continuous running

Question

I have flink batch job. What is the best way to run continuously? (It needs to restart when it's finished because the streaming job can provide new data)

I want to restart the job immediately if it's finished.

• Infinite cycle and inside call the tasks?
• Make a bash script and always push the job into the jobmanager? (I think it's a really big resource waste)

Thanks

Answer 1

In a similar use-case where we run Flink job against same collection; we trigger new job at periodic intervals. [daily, hourly etc.] https://azkaban.github.io/ can be used for scheduling. This is NOT really what you mentioned. But, a close-match which might be sufficient to solve your use-case.