CODEWITHSUNDEEP
javaapache-flinkflink-streamingflink-batch
MiniSu
MiniSu

Reputation: 590

How to keep flink batch job running continuously on local

I am practicing file reading through the flink batch processing mechanism on a Windows 10 machine.

I downloaded flink-1.7.2-bin-hadoop24-scala_2.12.tgz from flink's official site and executed start-cluster.bat .

I uploaded the jar though Flink's UI and was able to execute the job but the job finished in a matter of seconds.

I want to keep the job running continuously so that I can test my use case .

Can you guide my possible ways to achieve this?

Upvotes: 0

Views: 794

Answers (1)

David Anderson
David Anderson

Reputation: 43439

In Flink, batch jobs run until all of their input has been processed, at which point they have finished and are terminated. If you want continuous processing, then you should either

  • use some deployment automation (outside of Flink) to arrange for new batch jobs to be created as needed, or
  • implement a streaming job

In your case it sounds like you might be looking for the FileProcessingMode.PROCESS_CONTINUOUSLY option on StreamExecutionEnvironment.readfile -- see the docs for more info.

Upvotes: 1

Related Questions