CODEWITHSUNDEEP
integration-testingapache-flinkflink-streamingend-to-enddata-stream
Richard Deurwaarder
Richard Deurwaarder

Reputation: 2040

Apache Flink - End to End testing how to terminate input source

I've used apache flink in batch processing for a while but now we want to convert this batch job to a streaming job. The problem I run into is how to run end-to-end tests.

How it worked in a batch job

When using batch processing we created end-to-end tests using cucumber.

The problem in a streaming job

We would like to do something similar with the streaming job except the streaming job does not really finish.

So:

We could just wait 5 seconds after every test and assume everything has been processed but that would slow everything down a lot.

Question:

What are some ways or best practices to run end-to-end tests on a streaming flink job without forceable terminating the flink job after x seconds

Upvotes: 4

Views: 3721

Answers (1)

David Anderson
David Anderson

Reputation: 43499

Most Flink DataStream sources, if they are reading from a finite input, will inject a watermark with value LONG.MAX_VALUE when they reach the end, after which the job will be terminated.

The Flink training exercises illustrate one approach to doing end-to-end testing of Flink jobs. I suggest cloning the github repo and looking at how the tests are setup. They use a custom source and sink and redirect the input and output for testing.

This topic is also discussed a bit in the documentation.

Upvotes: 5

Related Questions