CODEWITHSUNDEEP
javamemory-managementconfigurationapache-kafkaout-of-memory
Seweryn Habdank-Wojewódzki
Seweryn Habdank-Wojewódzki

Reputation: 2167

How to calculate how much memory needs Kafka Broker?

How to calculate how much memory needs Kafka Broker having given troughput, average message size and retention time.

I am asking, as I see in our monitoring system, that after starting, Kafka broker lineary increases used JVM heap. At some point we got OutOfMemoryError.

We extend JVM Heap couple of times and the problem remains.

So it would be better not to guess, what Kafka needs by experimenting, but to calculate the value :-).

Upvotes: 1

Views: 2807

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26885

It's hard to calculate a precise value but you can use the rule of thumb from the Kafka documentation:

You need sufficient memory to buffer active readers and writers. You can do a back-of-the-envelope estimate of memory needs by assuming you want to be able to buffer for 30 seconds and compute your memory need as write_throughput*30.

Also since 1.0.0, you can use queued.max.request.bytes to limit how much memory brokers can use for incoming requests. That can help you avoid OOM and if you can monitor its usage with:

kafka.network:type=SocketServer,name=MemoryPoolAvailable
kafka.network:type=SocketServer,name=MemoryPoolUsed

That should allow you to determine how much memory your workload requires without too many issues.

Upvotes: 2

Related Questions