CODEWITHSUNDEEP
javaapache-kafkaheap-memory
lucas kim
lucas kim

Reputation: 910

Apache Kafka swap memory usage

I have a Kafka cluster of 10 machines.

Kafka's xmx configuration is now 8g.

However, after checking each Kafka process, I confirmed that 1g ~ 3g of swap memory is being used.

I know Kafka does not need to use a big heap.

In this situation, do I need to increase xmx of Kafka?

Or is there another solution?

In addition, the current swapiness setting for os is 1.

Upvotes: 1

Views: 2958

Answers (1)

mjuarez
mjuarez

Reputation: 16834

OS swapiness is a hard thing to nail down. I've had configurations where everything was running on less than half the memory available, and the OS still decided to swap out a few GB of memory to disk. I've also tried to completely turn off swapiness, and I saw no improvement at all on performance.

My recommendation is to leave OS swapiness alone. It's true that Kafka doesn't need a big JVM heap, but it will use any available off-heap RAM in the OS as page cache, to serve as much data/messages as possible from RAM. My recommendation would be to size the JVM using -Xms and -Xmx to the same value (8GB should be enough for most scenarios), and leave the rest of the available RAM for Kafka to use as off-heap memory, and for OS overhead.

Upvotes: 2

Related Questions