CODEWITHSUNDEEP
apache-kafkacouchbaseapache-kafka-connect
uzumas
uzumas

Reputation: 678

kafka connect and kafka host requirements

I am using Couchbase sink connector. CB and kafka are in 2 different EC2 instances, in different AWS regions. I am following these documentations:

  1. https://docs.couchbase.com/kafka-connector/current/quickstart.html
  2. https://kafka.apache.org/documentation/#connect_configuring

Based on these, I gather that connect must run on a host which also has kafka installed. Can my connect run on a remote host, such that I read from a remote kafka and sink messages into a remote CB bucket? Any documentation specific to this?

Also, I am getting following error:

        at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
    at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:118)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.3.0', encodedVersion=2.3.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}
    at org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:180)
    at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:152)
    at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:519)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:289)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)

Upvotes: 1

Views: 533

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191864

I gather that connect must run on a host which also has kafka installed.

Has Kafka libraries, yes, but not an actually running broker, no.

In other words, you cannot download Kafka Connect as a separarte service, so it therefore already comes with Kafka libraries when you get it.

Can my connect run on a remote host, such that I read from a remote kafka and sink messages into a remote CB bucket?

This is the recommended deployment. Similar to how you would not run normal producers and consumers on the brokers. Kafka Connect should be treated no different.

The bootstrap servers and couchbase connection strings would be their respective remote addresses

Docs are kinda here

Also, I am getting following error: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSinkConnector

If you look at the list of available connectors are:, you see yours is not listed. This means you didn't correctly load the plugin.path of the Connector properties with your library.

And as the Couchbase docs state:

Modify the plugin.path property to include the parent directory of kafka-connect-couchbase-<version>.jar

You can also verify your connector was loaded at all before posting a config by using the /connector-plugins endpoint

Upvotes: 5

Rilwan
Rilwan

Reputation: 88

Kafka Connect is not able to find your connector. Either you can set plugin.path to point to your connector jar file or you can copy the connector jar file into kafka lib directory.

Upvotes: 0

Related Questions