CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-platform
dennismonsewicz
dennismonsewicz

Reputation: 25542

Use Confluent Hub without Confluent Platform installation

I am using these installation instructions for getting confluent hub client https://docs.confluent.io/current/connect/managing/confluent-hub/client.html

But, when I get to the line to install the kafka-connect-elasticsearch connector

confluent-hub install confluentinc/kafka-connect-elasticsearch:latest

I continue to get this error message:

Unable to detect Confluent Platform installation. Specify --component-dir and --worker-configs explicitly.

Error: Invalid options or arguments

I have ElasticSearch and Kafka all installed via Homebrew on my Mac

Upvotes: 14

Views: 13971

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191738

Specify --component-dir and --worker-configs explicitly

You'll have to pass both options to include the Homebrew location of Kafka Connect's plugin.path setting (where Confluent Hub would install to) and your connect-distributed.properties (so that the plugin path can be setup if it isn't already)

e.g. (on a Mac, after brew install kafka)

confluent-hub install <name> \
   --component-dir /usr/local/Cellar/kafka/plugins \
   --worker-configs /usr/local/etc/kafka/connect-distributed.properties

This command should download and extract the connector zip into /usr/local/Cellar/kafka/plugins and modify the connect-distributed.properties file to include that path in its plugin.path property

Here is an example using a Kafka Connect docker image that I've built

Upvotes: 14

Rustam-Z
Rustam-Z

Reputation: 113

MacOS

In my case I have installed Kafka and Confluent Hub with brew and the installation path for Kafka is /usr/local/Cellar/kafka. And created a directory to install the connector.

Move to /usr/local/Caskroom/confluent-hub-client/6.2.1/bin, it is the directory confluent-hub is located.

Then I've executed

confluent-hub install confluentinc/kafka-connect-azure-blob-storage:1.6.2 \
  --component-dir /Users/Rustam_Zokirov/BigData/connectors \
  --worker-configs /usr/local/Cellar/kafka/3.0.0/libexec/config/connect-distributed.properties

Upvotes: 2

Related Questions