CODEWITHSUNDEEP
apache-kafka
jean
jean

Reputation: 2990

the behavior about num.partitions in properties file

I found the value of num.partitions is partition number on whole kafka cluster when auto topic was created.

I started 2 broker on my machine, brokerA and brokerB with its own server.properties (both num.partitons = 2), then I start a producer to write a message to kafka. I found each broker create 1 partition.

The problem is, when brokers in cluster each has its own different num.partitions, the logic of how many partitions of one topic will be obscure. For example, brokerA num.partiton=2, brokerB num.partitions=1. When a topic was auto created, how many partitions this topic will have?

I have tested this case but I can not find logical rules out. The behavior seems random, sometimes auto topic with 1 partition, sometimes 2.

Upvotes: 0

Views: 1995

Answers (1)

amethystic
amethystic

Reputation: 7089

By the design, the producer will choose a least-loaded broker to fetch topic metadata. For any non-existing topics, the chosen broker creates them with their own specified num.partitions if auto.create.topics.enable is true. By the least-loaded broker, it means the node with the fewest outstanding requests. In your test env, it might be brokerA sometimes but brokerB the other times. That's why you observe randomness.

Upvotes: 2

Related Questions