CODEWITHSUNDEEP
kubernetesapache-kafka-streams
Bala.vrad
Bala.vrad

Reputation: 53

How do I scale Kafka Streams app running on kubernetes

I have a kafka streams App which runs on a pod in Kube cluster.

Trying to find a way for Kube to scale up/down the pods based on kafka topic lag.

Has anyone walked the path before and willing to share some details ?.

Upvotes: 1

Views: 325

Answers (1)

Mahmoud Eltayeb
Mahmoud Eltayeb

Reputation: 436

For Scaling up based on the lag, your app should be able to publish metrics (Using micrometer for example) use meter type (Gauge https://micrometer.io/docs/concepts#_gauges)

Then you will need to collect the metrics using monitoring system (prometheus for example)

scale up/down based on the value of the meter/metric.

(Check this sample project as example https://github.com/learnk8s/spring-boot-k8s-hpa)

Note: But consider my comment as the max number of allowed consumers >= number of topic partitions

Upvotes: 1

Related Questions