CODEWITHSUNDEEP
apache-kafkareplication
Vms
Vms

Reputation: 209

Kafka reassignment of __consumer_offsets incorrect?

I am confused with how kafka-reassignment-paritions works for __consumer_offsets topic?

I start with 1 zk and 1 kafka broker, create a test topic with replication=1, partition=1. consume and produce. works fine.

I see __consumer_offsets topic created.

Now I add a second broker with, offsets.topic.replication.factor=2. I run the,

kafka-reassign-partitions --zookeeper zookeeper1:2181 --topics-to-move-json-file topics-to-move.json --broker-list "101,102" --generate

The generated reassignment does not look right. Only shows one replica even though there are 2 live brokers.

I was hoping to get following replicas for each partition: [101, 102] or [201, 101]

{
  "version": 1,
  "partitions": [
    {
      "topic": "__consumer_offsets",
      "partition": 19,
      "replicas": [101]
    },
    {
      "topic": "__consumer_offsets",
      "partition": 30,
      "replicas": [102]
    },
    {
      "topic": "__consumer_offsets",
      "partition": 47,
      "replicas": [101]
    }, ...

Appreciate any suggestion.

-Vms

Upvotes: 2

Views: 5480

Answers (2)

c0der512
c0der512

Reputation: 655

You can use the reassignment tool to generate json for your own advantage. In your topics-to-move.json, instead of putting partition values, put {"version":1, "topics":[{"topic":"__consumer_offsets"}]}

Now when you run kafka-reassign-partitions --zookeeper zookeeper1:2181 --topics-to-move-json-file topics-to-move.json --broker-list "101,102" --generate You will get recommended partition assignment from tool.

Save that file as reassignment-proposed.json and store old assignment into rollback.json.

Use that file to start reassignment /kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassignment-proposed.json --execute Verify assignment using --verify option

Upvotes: 1

amethystic
amethystic

Reputation: 7089

If you want to increase the replication factor for a topic, follow steps below:

  1. Create a json file containing the reassignment plan. In your case, the file might look like:

    {"version":1, "partitions":[
  {"topic":"__consumer_offsets","partition":0,"replicas":[101,102]}, 
  {"topic":"__consumer_offsets","partition":1,"replicas":[102,101]},
  {"topic":"__consumer_offsets","partition":2,"replicas":[101,102]},
  {"topic":"__consumer_offsets","partition":3,"replicas":[102,101]},
  ...
  {"topic":"__consumer_offsets","partition":49,"replicas":[101,102]}
]}

  2. Run command below to increase RF for this internal topic:

    bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassign.json --execute

Then run kafka-topics.sh --describe to see if replication factor is bumped up to 2.

Upvotes: 5

Related Questions