CODEWITHSUNDEEP
apache-flinkflink-streamingflink-statefun
Medivh
Medivh

Reputation: 1

Is there a way to broadcast configuration into all task managers or all FlatMapFunctions?

We currently have a flink-based streaming job (the task is composed of complex FlatMapFunctions DAG), and an http interface for fetching configuration.

Now I hope to read configuration from the http interface through a source function every 5 minutes with a parallelism of 1, and then distribute it to all task managers or FlatMapFunctions of the job. In FlatMapFunctions, the configuration will be read and will never not be changed.

I have read the documentationThe Broadcast State Pattern, but the method in the documentation seems to only apply to the first Function of the broadcast, and other subsequent downstream FlatMapFunctions cannot read the state of the broadcast. As shown in the figure below, only Co-Process-Broadcast can obtain the broadcast, but map func 1 and map func 2 cannot.

Broadcast state graph

1

Similar to QUESTION but different, I have many downstream FlatMapFunctions and expect them all to get the broadcast configuration.

Upvotes: 0

Views: 111

Answers (1)

kkrugler
kkrugler

Reputation: 9265

You can send the broadcast stream to multiple functions, so if your config state isn't big then that's likely what I'd do.

If the config state is very small (relative to the size of records being processed) then you could attach it to every incoming record in your BroadcastProcessFunction, so downstream operators have it in hand when processing each of their records.

Upvotes: 0

Related Questions