Initiated state change for partition from OfflinePartition to OnlinePartition failed

Question

I reconfigured my kafka cluster, changing:

default replication factor from 1 to 3 and also
changing the location of the kafka data dir on disk

So after restarting all nodes, the cluster seemed ok but then I noticed all the topics are failing to come online. In the logs there are messages like this for each topic:

state-change.log: [2018-02-01 12:41:42,176] ERROR Controller 826437096    epoch 19 initiated state change for partition [filedrop,0] from   OfflinePartition to OnlinePartition failed (state.change.logger)

So none of the topics are usable; Listing topics with kafkacat -L -b shows leaders not available.

Metadata for all topics (from broker -1: lol-045:9092/bootstrap):
7 brokers:
broker 826437096 at lol-044:9092
broker 746155422 at lol-047:9092
broker 651737161 at lol-046:9092
broker 728512596 at lol-048:9092
broker 213763378 at lol-045:9092
broker 622553932 at lol-049:9092
broker 746727274 at lol-050:9092
14 topics:
topic "lol.stripped" with 3 partitions:
 partition 2, leader -1, replicas:, isrs:, Broker: Leader not available
 partition 1, leader -1, replicas:, isrs:, Broker: Leader notavailable
 partition 0, leader -1, replicas:, isrs:, Broker: Leader not available

However, newly created topics are correctly replicated and healthy

topic "lol-kafka-health" with 3 partitions:
partition 2, leader 622553932, replicas: 622553932,213763378,651737161, isrs: 622553932,213763378,651737161
partition 1, leader 213763378, replicas: 622553932,213763378,826437096, isrs: 213763378,826437096,622553932
partition 0, leader 826437096, replicas: 213763378,746727274,826437096, isrs: 826437096,746727274,213763378

So I think some kind of metadata corruption happened during the reconfigure

My question is:

Is there any way I can get these topic partitions online again ?

Given that:

the broker ids were changed during the reconfigure
the zookeeper cluster for kafka went down temporarily during the reconfig

In addition, are there some procedures I can use to investigate how recoverable these topics are?

Many thanks in advance!

Initiated state change for partition from OfflinePartition to OnlinePartition failed

Answers (1)

Related Questions