CODEWITHSUNDEEP
apache-flinkflink-streaming
Bishamon Ten
Bishamon Ten

Reputation: 588

Flink: What does it mean to embed flink on other programs?

What does it mean to embed flink on other programs?
In the link here - https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/api_concepts.html#basic-api-concepts in second paragraph it says flink can be embedded in other programs.

I would like to know more about this. Like how to achieve it. A sample program would be very helpful. Using the above is it possible to achieve the following?

  1. Can we run flink programs as individual Actors?
  2. Can we route data between two flink programs?

Reason: I am asking the above two questions because my requirement is as below
I have some set of Flink Jobs/programs based on the config file I want only certain Flink Jobs/programs to process the input data and this keeps changing based on the config file. So there is a need for Flink jobs./programs(or the code in those jobs) to be always available and they need to pass data and communicate.

Kindly share your insights.

Upvotes: 1

Views: 1015

Answers (1)

Till Rohrmann
Till Rohrmann

Reputation: 13346

Running Flink embedded in other programs refers to Flink's local execution mode. The local execution mode runs a Flink program in your JVM. This entails that the job won't be executed distributedly.

What is currently not possible out of the box is to let Flink jobs control other Flink jobs. However, it is possible to build a Flink application which takes as input job descriptions and executes them. RBEA is an example of such a Flink application. The conceptual difference is that you don't have multiple Flink jobs but a single one which processes programs as input records.

Alternatively, you could take a look at Stateful functions which is a virtual actor framework built on top of Apache Flink. The idea is to provide a framework for building distributed stateful applications with strong consistency guarantees. With stateful functions, you would also build a single Flink application which processes events which could represent a form of computation.

Upvotes: 1

Related Questions