CODEWITHSUNDEEP
apache-flinkflink-streaming
Avinash Tripathy
Avinash Tripathy

Reputation: 601

In which scenario BroadcastConnectedStream in flink is really helpful?

In which scenario BroadcastConnectedStream in flink is really helpful?

A small example with clarification would be helpful.

Upvotes: 1

Views: 154

Answers (2)

kkrugler
kkrugler

Reputation: 9280

In addition to what David mentioned, if you have a keyed stream that you're connecting with the broadcast stream, then in your KeyedBroadcastProcessFunction's processBroadcastElement() method you can iterate over all of the keyed stream state, which isn't normally something you can do in a Flink operator. See The Broadcast State Pattern for more details.

Upvotes: 1

David Anderson
David Anderson

Reputation: 43707

I've written some examples that you can find here:

  1. https://github.com/ververica/flink-training-exercises/blob/master/src/main/java/com/ververica/flinktraining/examples/datastream_java/broadcast/BroadcastState.java
  2. https://training.ververica.com/exercises/nearestTaxi.html
  3. https://training.ververica.com/exercises/ongoingRides.html
  4. https://training.ververica.com/exercises/taxiQuery.html

In general, broadcast state is useful whenever you need to communicate something throughout the entire cluster. Most data sources are going to be partitioned, so that they can be processed in parallel by separate instances -- but some information is needed globally, like currency exchange rates, or thresholds, or machine learning models. If that globally useful data is static, you can simply load it from a file, but if it needs to be updated dynamically during runtime, then using a broadcast stream makes sense.

Upvotes: 2

Related Questions