CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Parkash kumar
Parkash kumar

Reputation: 31

How to understand kafka streams aggregation?

I am new to kafka and learning it. I am just working on aggregating data for employees but running into issues. Can someone please help.

I have a topic timeoffs with key time_off_id and value of type object which also contain employee id. So I want to build a store where employee id should be the key and value should be list of that employee's time offs.But I am following below approach but running into issue. While aggregating the data, it is saying bad return type in method reference: can not convert ArrayList to VR. Can you help me.

Code:

KTable<String, TimeOff> timeoffs = builder.table(topic);
KGroupedTable<String, TimeOff> groupedTable = timeoffs.groupBy(
    (key, value) -> KeyValue.pair(value.getEmployeeId(), value)
);
groupedTable.aggregate(ArrayList<TimeOff>::new, (k, newValue, aggValue) -> {
  aggValue.add(newValue);
  return aggValue;
}, Materialized.as("NewStore"));

I also tried this approach, but again this did not resolve the issue.

TimeOffList class:

package com.kafka.productiontest.models;

import java.util.ArrayList;

public class TimeOffList {
  ArrayList list = new ArrayList<TimeOff>();

  public TimeOffList add(Object s) {
    list.add(s);
    return this;
  }
}

In streaming class:

groupedTable.aggregate(TimeOffList::new,
    (k, newValue, aggValue) -> (TimeOffList) aggValue.add(newValue));

After implementing your solution, this issue gone but now facing issue with serde. I have implemented TimeOffListSerde. Please check below code

KStream<String, TimeOff> source = builder.stream(topic);
source.groupBy((k, v) -> v.getEmployeeId())
    .aggregate(ArrayList::new,
        (key, value, aggregate) -> {
          aggregate.add(value);
          return aggregate;
        }, Materialized.as("NewStore").withValueSerde(new TimeOffListSerde(TimeOff.class)));

TimeOffListSerde.java

package com.kafka.productiontest.models;

import org.apache.kafka.common.serialization.Deserializer;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.Serializer;

import java.util.ArrayList;
import java.util.Map;

public class TimeOffListSerde implements Serde<ArrayList<TimeOff>> {
  private Serde<ArrayList<TimeOff>> inner;

  public TimeOffListSerde() {
  }

  public TimeOffListSerde(Serde<TimeOff> serde){
    inner = Serdes.serdeFrom(new TimeOffListSerializer(serde.serializer()), new TimeOffListDeserializer(serde.deserializer()));
  }

  @Override
  public void configure(Map<String, ?> configs, boolean isKey) {
    inner.serializer().configure(configs, isKey);
    inner.deserializer().configure(configs, isKey);
  }

  @Override
  public void close() {
    inner.serializer().close();
    inner.deserializer().close();
  }

  @Override
  public Serializer<ArrayList<TimeOff>> serializer() {
    return inner.serializer();
  }

  @Override
  public Deserializer<ArrayList<TimeOff>> deserializer() {
    return inner.deserializer();
  }
}

Upvotes: 0

Views: 2527

Answers (1)

bistros
bistros

Reputation: 1179

would want you this ?

KStream<String, TimeOff> source = builder.stream(sourceTopic);
KTable<String, List<TimeOff>> table = source.groupBy((k, v) -> v.getId())
    .aggregate(ArrayList::new,
            (key, value, aggregate) -> {
                aggregate.add(value);
                return aggregate;
            }, Materialized.as("NewStore"));

Upvotes: 5

Related Questions