CODEWITHSUNDEEP
apache-kafka
Karthikeyan Rasipalay Durairaj
Karthikeyan Rasipalay Durairaj

Reputation: 2338

Increase the Number of partitions

We are working on Confluent Platform and we are still getting to know the internals. But we have implemented generic use cases . We are trying to optimizing our cluster

In my use case, I need to increase the number of partitions of a topic . What is the impact of it ? Can you please share of it

Upvotes: 0

Views: 198

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

Sure, you can increase partitions.

However,

  1. Increasing partitions does not move existing data. If using Confluent Enterprise, you could use confluent-rebalancer, or if not, then kafka-reassign-partitions CLI tool. So, you'll definitely want to rebalance a topic to "optimize" the cluster.
  2. During the retention period of the topic (read: for the existing data), if you previously had a producer sending data to partition N, and now had N+1 partitions, then you lose ordering of those messages that solely existed in partition N. New messages could be spread across new partitions that a new producer calculates with the DefaultPartitioner. If you don't send keys with messages, then this isn't a problem.

Upvotes: 1

Related Questions