CODEWITHSUNDEEP
apache-flinkflink-streaming
FlinkNoob
FlinkNoob

Reputation: 141

How can I sum multiple fields in Flink?

I want to get the sum of multiple fields. I use this code to explain my pain:

 // parse the data, group it, window it, and aggregate the counts
 val windowCounts = text
      .flatMap { w => w.split("\\s") }
      .map { w => WordWithCount(w, 1, 2) }
      .keyBy("word")
      .timeWindow(Time.seconds(5), Time.seconds(1))
      .sum("count")

case class WordWithCount(word: String, count: Long, count2: Long)

I want the sum of both fields (count and count2) in my time windows. I can't add multiple sums like that:

 val windowCounts = text
        .flatMap { w => w.split("\\s") }
        .map { w => WordWithCount(w, 1, 2) }
        .keyBy("word")
        .timeWindow(Time.seconds(5), Time.seconds(1))
        .sum("count", "count2")

I have no idea how to do it.

Upvotes: 2

Views: 3240

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18987

The DataSteam API does not offer built-in operators to sum multiple fields.

There are two options:

  1. Implement a custom ReduceFunction that sums both fields.
  2. Have a look at Flink's Table API or SQL support. Both can perform multiple aggregations on a group window.

Upvotes: 2

Related Questions