CODEWITHSUNDEEP
c#.netapache-kafkakafkaflow
James
James

Reputation: 541

KafkaFlow Batch Processing Large Batches Not Working as Expected

I have a .NET app that is using a Batch Consumer with the KafkaFlow library. I am trying to consume large batches of messages (250K) and insert them into a database. I seem to have a configuration issue as I have a couple of problems.

  1. The batch only contains about half the number of messages I expect. I am consistently pulling between 100K and 105K messages even though there is a lag of 2M+ records on the topic. Why is this occurring?
  2. It seems that even with this number of records once the process completes, there is a significant gap (3-5 minutes) from when the process finishes and the next batch is consumed.

Again, I'm almost sure these are configuration issues with how I have the consumer setup. Any input would be appreciated.

Upvotes: 1

Views: 75

Answers (0)

Related Questions