CODEWITHSUNDEEP
apache-kafkakraft
musmil
musmil

Reputation: 53

Kafka raft brokers seems to be not recording new active controller on leader change

edit: kafka version 3.5.1 with separated kraft quorum, with 3 members.

Long story short, we recently updated the java versions and jmx agents on our kafka clusters. Which required a rolling restart of all components of our kafka cluster. Everything went smooth, we saw no problems at all. We restarted brokers first, then the controllers. Which means the active controller changed for the Kraft controller quorum.

Until now everything is ok, today (3 days after the restarts) we randomly see the below errors saying the brokers are recording the new controller. This cluster is not massively in use and can have brokers not doing much for days. These logs doesnt really cause any problems, they happend once and didn't effect anything, but they are still annoying to see.

My question here is, on quorum leader change aren't brokers recording the active controller immediately? Do they need to have trigger which causes them to get active controller to realize which controller is the leader? Isn't there a mechanism to get metadata updates for brokers about quorum state?

2025-02-20 09:45:39.428 
[2025-02-20 09:45:39,368] INFO [BrokerToControllerChannelManager id=106 name=forwarding] Node 101 disconnected. (org.apache.kafka.clients.NetworkClient)
2025-02-20 09:35:39.237 
[2025-02-20 09:35:39,041] INFO [broker-106-to-controller-forwarding-channel-manager]: Recorded new controller, from now on will use node ***:9093 (id: 101 rack: null) (kafka.server.BrokerToControllerRequestThread)
2025-02-20 09:35:39.237 
[2025-02-20 09:35:39,040] INFO [BrokerToControllerChannelManager id=106 name=forwarding] Client requested disconnect from node 103 (org.apache.kafka.clients.NetworkClient)



2025-02-20 09:34:42.352 
[2025-02-20 09:34:42,113] INFO [BrokerToControllerChannelManager id=108 name=forwarding] Node 101 disconnected. (org.apache.kafka.clients.NetworkClient)
2025-02-20 09:24:42.178 
[2025-02-20 09:24:42,045] INFO [broker-108-to-controller-forwarding-channel-manager]: Recorded new controller, from now on will use node ***:9093 (id: 101 rack: null) (kafka.server.BrokerToControllerRequestThread)
2025-02-20 09:24:42.178 
[2025-02-20 09:24:42,043] INFO [BrokerToControllerChannelManager id=108 name=forwarding] Client requested disconnect from node 103 (org.apache.kafka.clients.NetworkClient)

Regards.

Upvotes: 0

Views: 31

Answers (0)

Related Questions