CODEWITHSUNDEEP
amazon-web-servicesapache-kafkaaws-lambdaaws-event-bridge
Dhanendra
Dhanendra

Reputation: 113

Using Self hosted Kafka as event source for AWS Lamda

I want to use a self-hosted (in EC2 instance) Kafka stream to use as an event source to trigger the lambda function. Although lamda can easily be configured(here) to get triggered from a Kafka stream, I don't want to trigger for all the messages but for only a few conditions.

We can use pattern matching in Eventbridge but can we use self-hosted Kafka as a source there? If yes, then how?

Upvotes: 2

Views: 754

Answers (1)

Yayotrón
Yayotrón

Reputation: 1889

You need to implement your custom filtering somewhere. I think your best bet would be to use Kafka Streams to find the relevant messages which you want to act upon and then send them to a separate topic. Here's an article explaining how this can be implemented.

Other alternative solutions I could suggest would be to use AWS Kinesis or a custom kafka consumer, nevertheless they all follow the same concept:

  1. Process the stream.
  2. Find the relevant messages.
  3. Send them to some other place (e.g another kafka topic) which will be the input for your lambda.

Upvotes: 2

Related Questions