CODEWITHSUNDEEP
javaspring-bootapache-kafkaspring-kafka
ppb
ppb

Reputation: 2633

Spring Boot kafkaTemplate consumer message load and processing message

in my application I am using Spring Boot kafkaTemplate for consuming the messages. I am new to kafka with Spring Boot. I have added a consumer as below -

 @KafkaListener(topics = "#{'${app.kafka.consumer.topic}'.split(',')}")
 public void receivedMessage(ConsumerRecord<String, String> cr, @Payload String message){
    log.info("Message received from topic {} ", cr.topic());
    //TODO
}

On a topic we will receive near about 200K messages per second. Ones I received message will send to another method for processing which filtered the messages based on certain criteria and then publishing the filtered message to another topic.

My question is, does above @KafkaListener method will handle this load or do I need to do any special treatment like threading or concurrency.

Upvotes: 0

Views: 779

Answers (1)

Gary Russell
Gary Russell

Reputation: 174729

It depends entirely on your workload, the number of cores in your CPU, etc, etc. In general, increasing the concurrency will provide more throughput (as long as you have at least that many partitions on the topic).

But, your downstream code must be completely thread-safe.

Even then, there may be other bottlenecks in your code (DB etc).

If all you are doing is computation, transformation, and publishing to another topic, without doing any other I/O, increasing the concurrency should definitely help.

The only real solution is experimentation and, if you don't get the throughput you need, profile your application.

You can't learn such skills without actually doing it.

Upvotes: 1

Related Questions