CODEWITHSUNDEEP
apache-kafka
Rj1
Rj1

Reputation: 469

Why __consumer_offsets topic in kafka is not spreading to all the brokers?

I have a 3 zk nodes cluster. And 7 kafka broker nodes.

So when I create any topic then I can set replica factor and no of partitions using command line arguements.

And these partitions are spread to all the 7 brokers.But there is one topic i.e. __consumer_offsets,it is created automatically and it is spreading to only 1 broker(id=0) with 1 replication.

I am aware that I could change the default no of replication factor in config file.

But, this topic is replicated to only one broker. So what is the parameter which could be changed to replicate the partitions to all the brokers.

Upvotes: 9

Views: 14388

Answers (3)

Liju John
Liju John

Reputation: 1896

You need to set the "offsets.topic.replication.factor" to the desired number of replicas in the broker config.

The replication factor for the offsets topic (set higher to ensure availability). To ensure that the effective replication factor of the offsets topic is the configured value, the number of alive brokers has to be at least the replication factor at the time of the first request for the offsets topic. If not, either the offsets topic creation will fail or it will get a replication factor of min(alive brokers, configured replication factor)

Default: 3
Importance: High

Reference - http://kafka.apache.org/documentation/#brokerconfigs

Upvotes: 2

ppatierno
ppatierno

Reputation: 10075

It sounds really strange to me. I deployed a 3 Kafka brokers cluster and my __consumer_offsets topic has 50 partitions (the default) split across the 3 brokers and having 3 (the default) as replicas. What you are describing could happen when you start with a single Kafka broker, create consumers for reading topics so that the __consumer_offsets is automatically created and it will be just on the broker 0 (the only one). After that, you add new Kafka brokers: without doing anything the __consumer_offsets will still stay on broker 0; you need to use Kafka a manual partition reassignment for doing that as mentioned above.

Upvotes: 10

HansHarhoff
HansHarhoff

Reputation: 2007

You will need to use Kafka partition reassignment. This is unfortunately a semi manual process currently: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

Upvotes: 3

Related Questions