CODEWITHSUNDEEP
apache-flinkapache-stormflink-streaming
shanker861
shanker861

Reputation: 629

How to set a Flink Operator acting on many streams

I am looking into using Flink for streaming Engine. I am coming from apache-storm and As I understand storm's Bolt is similar to Flink's task/operator. In storm one can have

 builder.setBolt("TEST", new TestBolt(),5)
        .fieldsGrouping("Source1", "ID1")
        .fieldsGrouping("Source2","ID2)
        .fieldsGrouping("Source3","ID1")
        .allGrouping("Source4");

How can I achieve something similar with Flink. Basically I want my test Bolt to have state from Source2, source3, source4 and do some computation when data from source 1 comes.

Upvotes: 0

Views: 82

Answers (1)

David Anderson
David Anderson

Reputation: 43707

Your options for combining streams in Flink include union (for merging n streams of the same type), connect for jointly processing two streams of any type with a CoFlatMap or CoProcessFunction, and broadcast.

In some cases it's preferable to build a sort of binary tree, connecting, for example, streams 1 and 2 to form stream 12, and separately, streams 3 and 4 to create stream 34, and then connect stream12 with stream 34.

The alternative is to create some sort of union type that can hold an object from any of the streams, and then use union to merge the streams. Flink includes an Either type that can be helpful in these situations.

Upvotes: 1

Related Questions