CODEWITHSUNDEEP
apache-kafkadockerfileapache-kafka-connect
LeniM
LeniM

Reputation: 21

Kafka connect in docker

I am trying to set up a Dockerized environment running kafka-connect and having a standalone connector (s3-sink).

I am using docker-compose and Kafka, zookeeper and my producer all works beautifully, but the docker image for kafka connect shuts down immediately after booting. I used examples from the documentation, but none uses a ENTRYPOINT or CMD.

Further more, when I prevent it to shutdown and try to launch my connector with ./bin/connect-standalone it fails with the following error org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.s3.S3SinkConnector

My Docker compose is literraly two lines, the first one should be doing what my first problem is, the second is installing what the second issue claims is missing...

Surely I am missing something here in how this is supposed to be working, any idea ?

I prevented the image to shutdown, and it still does not expose anything on port 8083 with kafka-connect REST interface 

FROM confluentinc/cp-kafka-connect-base:5.3.0
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:5.3.0
COPY . .
#CMD["connect-standalone", "connect-standalone.properties", "s3-sink.properties"]

I expect the Docker image to not shutdown and expose port 8083

Answer : the rest API is started when a connector is launched. If the connector fails, the api is shut down so will the container.

Upvotes: 0

Views: 2834

Answers (2)

Daniel Nguyen
Daniel Nguyen

Reputation: 13

You'll need to set the environment variable CONNECT_PLUGIN_PATH to whatever path the plugin is getting installed to.

If you use --verbose on the confluent-hub install command,the plugin is being installed to /usr/share/confluent-hub-components/

The executable is looking in a different folder by default.

Upvotes: 1

LeniM
LeniM

Reputation: 21

The rest API is started when a connector is launched. If the connector fails, the api is shut down so will the container.

Upvotes: 0

Related Questions