CODEWITHSUNDEEP
linuxapache-kafka
jessica
jessica

Reputation: 2610

kafka + how to spread topics from one disk to all other new disks

We have Kafka cluster version 2.7

And each machine was with one disk as the following

/dev/sdb                82% /var/kafka/data1

And all topics are under /var/kafka/data1

We decided to add 7 disks new disks to each kafka as the following

And we created XFS file-system on each new disk include mount ( and fstab configuration )

as the following :

/dev/sdb                     82% /var/kafka/data1
/dev/sdc                      1% /var/kafka/data2
/dev/sdd                      1% /var/kafka/data3
/dev/sde                      1% /var/kafka/data4
/dev/sdf                      1% /var/kafka/data5
/dev/sdg                      1% /var/kafka/data6
/dev/sdh                      1% /var/kafka/data7
/dev/sdi                      1% /var/kafka/data8

We configured the server.properies on each kafka And then we restart kafka service on all machines

So we are now after Kafka restart ,

Since we have additional 7 disks , we want topics that located /var/kafka/data1 , will separated to all other new disks

But this isn't the situation after Kafka brokers restart and all mount point folder except - /var/kafka/data1 , are without topics

So the Question is – what is the procedure or process to fill the new disks with the topics , ?

Upvotes: 0

Views: 419

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191894

To my knowledge, Kafka doesn't have an intra node rebalance like Hadoop has.

In theory, you could try shutting down the broker, mounting the new partitions to a directory, adding those to log.dirs, then moving the topic partitions over to those directories, and restarting the broker. However, I'm not sure how you'd handle files like meta.properties

Another alternative, after updating the broker config, would be to use the reassign partition tool to completely move partitions or topics off that larger broker, then move it back. Or just completely start deleting directories from the large usage disk, and let replication move it back across all disks, hopefully

Upvotes: 1

Related Questions