Handling Kafka Consumer Lifecycle When Scaling to Zero with KEDA in Kubernetes

Question

I'm working on a system where Kafka is used to distribute tasks to background workers running in a Kubernetes (K8s) environment. These workers scale down to zero when there’s no lag in the Kafka topics, and we use KEDA to monitor Kafka lag and scale the workers back up when traffic arrives.

The Problem: When the workers scale down to zero, there are no active consumers in the Kafka consumer group. After the offset retention period expires, Kafka marks the consumer group as dead, which causes problems when new messages arrive later.

Current Workaround: To prevent the consumer group from being deleted, we’ve implemented a heartbeat cron job that sends a dummy message to all Kafka topics once a day. This keeps the consumer group alive but feels like a hack.

What I'm Looking For: I’d like to find a better solution that avoids sending dummy messages while ensuring that the consumer group remains active when the workers are scaled to zero.

Is there a more efficient way to handle this scenario? Any best practices or alternative approaches would be greatly appreciated!