CODEWITHSUNDEEP
apache-flinkflink-streaming
jag
jag

Reputation: 93

why flink stops my stream application?

My code uses readTextFile to read log files, and when i run the jar in Flink (/opt/flink-1.0.3/bin/flink run -m yarn-cluster -yn 2 /home/flink/flink-json-0.1.jar), it processes successfully the lines inside and stops my application instead of being waiting for new lines. Do i need some param for doing it?

val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream = env.readTextFile("hdfs:///test/ignicion.io")

Thank you in advance

Upvotes: 0

Views: 271

Answers (1)

snntrable
snntrable

Reputation: 921

You are looking for

StreamExecutionEnvironment.readFileStream(String filePath, long intervalMillis, WatchType watchType)

For the WatchType you have the following options

  • ONLY_NEW_FILES,
  • REPROCESS_WITH_APPENDED,
  • PROCESS_ONLY_APPENDED;

The stream from 

StreamExecutionEnvironment.readTextFile(String filePath, String charsetName)

will be finished after reading all files. I think, it is mainly for local testing during development.

Upvotes: 2

Related Questions