CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Jiayi Liao
Jiayi Liao

Reputation: 1009

Why is Kafka Consumer's Performance with latest offset and earliest offset different?

The fetch request is the same. 

maxWait: 100ms
minBytes: 1
fetchSize: 1048576

When I consume from the earliest offset, it's very fast and can be 10,000 records per second. However, when I consume from the latest offset, it's very slow and can be 10,000 records per minute.

Anyone knows why?

Upvotes: 0

Views: 889

Answers (1)

Soheil Pourbafrani
Soheil Pourbafrani

Reputation: 3427

In the earliest mode, Kafka has data stored on the disk and in every fetching data it can fetch 1048576KB of data (untill reading all the data on the disk), but in the latest mode Kafka listen for new data and it also depends on the producer rate of data generation. In the latest mode Kafka will wait maxWate and received all new data. So it's a natural behavior.

Upvotes: 1

Related Questions