CODEWITHSUNDEEP
apache-flinkflink-streaming
Richard Grossman
Richard Grossman

Reputation: 109

How to process this calculation with Flink

I would like to make a real-time filter using Flink. the idea is to have a value by key stored as accumulator and to calculate a ratio versus the total sum for all keys.

I know it's impossible to share state between keyed operator thus I'm not able to calculate the total value

example :

  1. k1,1
  2. k2,3
  3. k1,1
  4. k2,5
  5. k3,0

I need to calculate on the stream the following ratio 1/1 , 3/4, 2/5, 8/10, 0 (is always filtered) etc...

Thanks for help

Upvotes: 1

Views: 274

Answers (1)

Alex
Alex

Reputation: 859

Create a custom stateful operator with the following state:

int totalSum;
Map<Key,Ratio> map;

Every event increments the total sum, then update the map according to the event key.

Example:

After 1st event k1,1 your state is: 

totalSum 1
map 
   k1, 1/1

And you emit the event: k1, 1/1

======

After 2nd event k2,3 your state is: 

totalSum 4
map 
   k1, 1/1
   k2, 3/4

And you emit the event: k2, 3/4

[.. continue]

Upvotes: 1

Related Questions