CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-partitionrebalancing
Jim
Jim

Reputation: 4425

Does a kafka broker keep persistent tcp connections with all of the consumers in all the groups?

So Kafka client's for consuming messages uses poll() which along with the heartbeat is what makes the broker decide if a client is still alive.
What is not clear to me is if the poll() and heartbeat are sent over persistent network connections or each poll() or heartbeat are creating a new tcp connection each time.
I would assume the connections would be persistent but would that mean that for a broker with several thousands of consumers there would be several thousands of open tcp connections (most of them potentially idle if the consumer is doing some event processing)?

Upvotes: 1

Views: 29

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26895

Consumers keep persistent connections to all brokers they are fetching data and keep a separate connection to their consumer group coordinator, even if it's also a broker they are already fetching data from. In the node metrics the connection to the coordinator has the node attribute set to -1.

In most cases establishing a new connection is expensive, so Kafka clients try to reuse existing connections and keep connections they are actively using.

Connections that are idle are automatically closed after connections.max.idle.ms which is 9 minutes by default.

Consumers use the coordination connection to send their heartbeats. While heartbeats are small, they happen every heartbeat.interval.ms which is 3 seconds by default.

So yes, if you have thousands of consumers (or other Kafka clients) connected to your cluster, each broker can have thousands of network connections.

It's often recommended to monitor the number of connections (and the creation, closure rate) of your clusters. Both Kafka clients and brokers expose these metrics. See https://kafka.apache.org/documentation/#selector_monitoring

Upvotes: 1

Related Questions