CODEWITHSUNDEEP
spring-bootspring-kafka
SoftwareSavant
SoftwareSavant

Reputation: 9737

multiple Spring Kafka Consumer one class multiple groups

I am experimenting with Apache Kafka and Spring Boot.

Is there a way to set up multiple Listeners in one JVM, Each of them associated to a different group (so that they each respond to an event) on a topic?

Is there a good pattern for this?

@KafkaListener(topics ="${slappy.consumer.topic}", groupId = "t1, t2, t3, t4, t5, t6, t7, t8, t9, t10")
public void listenToGroup(@Payload WackyMessage message) {

    log.info("Wacky Consumer with processItemId of received message: '{}'", message);

    process(message, ProcessItemIdentifiers.omni_silo0_deleter);
}

Sounds like this is the sort of thing Akka is for. I was hoping I could just do naked spring. I am not sure If I want to add Akka into the mix at this point.

Upvotes: 0

Views: 814

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121177

Your possible groupId = "t1, t2, t3, t4, t5, t6, t7, t8, t9, t10" solution is logically wrong. You still have the same listener method, so what is the point to call it so many times for the same record in Kafka topic?

If you would have different @KafkaListener methodd signatures, I would agree with you about different groups on them, so each of them would really receive the same message and it could be done in parallel as well, since each @KafkaListener blows up its own listener container process.

On the other hand you can leverage an @EventListener abstraction from Spring to distribute a single record from a single @KafkaListener to any possible number of subscribers to that event. In this case you don't need to think about different groups: the single one is enough to consumer from Kafka and process everything rest in the target application already.

Upvotes: 1

Related Questions