CODEWITHSUNDEEP
dockerdocker-composeapache-kafka-connectaws-msk
Sultan of Swing
Sultan of Swing

Reputation: 468

Transition from Kafka Connect (Docker based) to AWS MSK Connect

I am using this repo as a reference, and I've successfully got it up and running locally: https://github.com/rhaycock/Kafka-Connect-POC/tree/master/kafka-connect-main

I'm trying to set up a connector in MSK Connect and I am using this sink spec: https://github.com/rhaycock/Kafka-Connect-POC/blob/master/kafka-connect-main/config/sink/gcs-connector.json

There are also a lot of environment variables in the docker-compose.yml file: https://github.com/rhaycock/Kafka-Connect-POC/blob/master/kafka-connect-main/docker-compose.yml

My question at this point, is how do I get all of these variables from the Docker compose file into an MSK connector? Do they go in the "Connector Configuration" in MSK? Or the Worker Configuration? Or somewhere else? Specifically I need ones like CONNECT_GROUP_ID, CONNECT_CONFIG_STORAGE_TOPIC, CONNECT_OFFSET_STORAGE_TOPIC, CONNECT_STATUS_STORAGE_TOPIC, among a few others.

Upvotes: 0

Views: 661

Answers (1)

jorwy
jorwy

Reputation: 21

The majority of environment variables in the docker-compose.yml file you linked to are managed by the MSK Connect service

When using MSK Connect you have control over properties in the following locations:

  1. Connector configuration
  2. Worker configuration (provided the properties in the defined allow-list here)

Other properties, such as those which dictate how the Connect framework is configured (e.g KAFKA_CONNECT_MODE in the docker-compose file) are managed by MSK Connect.

Considering the docker-compose.yml file you linked, the list below maps the env vars to properties that are customizable in MSK Connect. Properties excluded from this list are currently service managed:

  • KAFKA_BOOTSTRAP_SERVERS -> kafkaCluster.apacheKafkaCluster.bootstrapServers in the CreateConnector request.
  • AWS_ROLE_ARN -> serviceExecutionRoleArn in the CreateConnector request.
  • CONNECT_OFFSET_STORAGE_TOPIC -> offset.storage.topic in a Worker Configuration resource, learn more here.

One final note, the README.md of the repository you linked also states:

Copy any required jar files required for CLASSPATH and update the .env with the correct details

In MSK Connect you can achieve this by creating a Custom Plugin resource with the JARs you want to bundle (e.g your connector implementation)

Upvotes: 2

Related Questions