CODEWITHSUNDEEP
apache-kafkakafka-consumer-apiapache-kafka-streams
Nikunj Darji
Nikunj Darji

Reputation: 23

Kafka Topic Lag is keep increasing gradually when Message Size is huge

I am using the Kafka Streams Processor API to construct a Kafka Streams application to retrieve messages from a Kafka topic. I have two consumer applications with the same Kafka Streams configuration. The difference is only in the message size. The 1st one has messages with 2000 characters (3KB) while 2nd one has messages with 34000 characters (60KB).

Now in my second consumer application I am getting too much lag which increases gradually with the traffic while my first application is able to process the messages at the same time without any lag.

My Stream configuration parameters are as below,

application.id=Application1
default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
num.stream.threads=1
commit.interval.ms=10
topology.optimization=all

Thanks

Upvotes: 2

Views: 3322

Answers (1)

Vasyl Sarzhynskyi
Vasyl Sarzhynskyi

Reputation: 3955

In order to consume messages faster, you need to increase the number of partitions (if it's not yet done, depending on the current value), and do one of the following two options:

1) increase the value for the config num.stream.threads within your application

or

2) start several applications with the same consumer group (the same application.id).

as for me, increasing num.stream.threads is preferable (until you reach the number of CPUs of the machine your app runs on). Try gradually increasing this value, e.g go from 4 over 6 to 8, and monitor the consumer lag of your application.

By increasing num.stream.threads your app will be able to consume messages in parallel, assuming you have enough partitions.

Upvotes: 1

Related Questions