CODEWITHSUNDEEP
kubernetesapache-kafka
sigma1510
sigma1510

Reputation: 1281

Triggering kubernetes job for a kafka message

I have a kubernetes service that only does something when it consumes a message from a Kafka queue. The queue does not have messages very often, and running the service as a job triggered whenever a message is found would save resources.

I see that Kubernetes has this functionality for AMQP-type message services: https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/

Is there a way to adapt this for Kafka, given that Kafka does not support AMQP? I'd switch to a different messaging system, but I have other services that also read from this queue that require Kafka.

Upvotes: 1

Views: 1647

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191710

That Kafka consumer Service is all you really need. If you want to save resources, this could be paired with KEDA autoscaler such that it scales up and down, depending on load or consumer group lag.

Or you can use serverless platforms such as KNative to trigger based on Kafka (or other messaging systems) events.

Kafka does not support AMQP

Kafka Connect should be able to bridge AMQP to Kafka. E.g. Apache Camel has connectors for both.

Upvotes: 2

Related Questions