CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Akash Sethi
Akash Sethi

Reputation: 2294

Kafka Reduce Lag for Consumer

I am setting up the new Kafka cluster and for testing purpose I created the topic with 1 partition and 3 replicas.

Now, I am firing the messages via producer in parallel say 50K messages per Second. And I have create One Consumer inside a Group and its only able to fetch 30K messages per second.

I can change topic level, partition level, consumer level configurations.

I am monitoring everything via grafana + prometheus.

Any Idea which configuration or something else can help me to consumer more data??

Thanks In Advance

Upvotes: 4

Views: 5082

Answers (1)

senseiwu
senseiwu

Reputation: 5259

A Kafka consumer polls the broker for messages and fetches whatever messages are available for consumption, depending upon the consumer configuration used. In general, it is efficient to transfer as much data is possible in a single poll request if increasing throughput is your aim. But how much data is transferred in a single poll is determined by the size of messages, number of records and some parameters which control how much time to wait for messages to be available.

In general, you can influence throughput using one or more of the following consumer configurations:

fetch.min.bytes

max.partition.fetch.bytes

fetch.max.bytes

max.poll.records

fetch.max.wait.ms

Upvotes: 7

Related Questions