CODEWITHSUNDEEP
apache-kafkaaws-lambdaapache-kafka-connect
Ravishankar Chaubey
Ravishankar Chaubey

Reputation: 43

Kafka Connect AWS Lambda Sink

How does this connector plugin works? Kafka Connect AWS Lambda Sink

In docker-compose.yml file it is pulling docker images of zookeeper, kafka & kafka-connect from confluentinc repository.

Can we utilize this plugin only for connecting aws lambda as a kafka consumer with kafka cluster deployed separately on ec2 instances.

Can we consume this connector plugin without the use of kafka connect?

Upvotes: 4

Views: 3559

Answers (2)

lenny
lenny

Reputation: 81

You can use the plugin with a separately deployed Kafka cluster. Either deploy the connector jar to your Kafka Connect cluster or run the Kafka Connect container in Docker and deploy the lambda connector there. You just need to modify the docker-compose.yml configuration of Kafka Connect to point to you Kafka cluster. There’s a potentially better example in github.com/llofberg/kafka-connect-rest repository.

Disclaimer: I wrote the connector.

Upvotes: 2

Robin Moffatt
Robin Moffatt

Reputation: 32140

Can we consume this connector plugin without the use of kafka connect?

No. Kafka Connect connectors run on Kafka Connect.

But, Kafka Connect is part of Apache Kafka, and gives you a runtime environment which handles things like node failure, scaleout, centralised config & management—so it's actually a good thing to use. From a user's point of view it's 'just' config files that you need to write to use it.

You can read more about Kafka Connect here

Disclaimer: I work for Confluent.

Upvotes: 2

Related Questions