CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
stephen fernandes
stephen fernandes

Reputation: 133

Too much load on kafka partition (extra load on single partition)

Suppose, there is trade capture application. The application consumes message via kafka. The partition id is stock-id (eg google, apple, tesla). This works fine in normal days. Suppose there is bad news for company and people are selling stocks for X company. Then in this case, all the messages would come to single partition during that trading session or day.
How do i handle, this efficiently? Can we apply multiple consumers to same partition?
its due to overloaded partition on random day. We have more than dozens of partitions along with dozens of consumers. All the partitions/ consumers are equally distributed everytime throughout the year. Its when there is sudden spike of data in single partition which happens once in month or quarterly .

Upvotes: 0

Views: 923

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191973

Can we apply multiple consumers to same partition

Not in the same consumer group, no.

The only way to reasonably handle this is to increase max.poll.records and other consumer properties to consume faster from that partition, and/or all partitions. Unfortunately, you won't know ahead of time which partition will get "overloaded".

The other alternative is to redesign your topic(s) such that "stock tickers" are not your partition ID and whatever you do choose as your partitioning strategy is not driven by end-user behavior that is out of your control (or otherwise define your own Paritioner class).

Upvotes: 1

Related Questions