CODEWITHSUNDEEP
javamongodbapache-kafka
user1575148
user1575148

Reputation: 589

ClassNotFoundException: com.mongodb.ConnectionString for Apache Kafka Mongodb connector

I am configuring a Kafka Mongodb sink connector on my Windows machine.

My connect-standalone.properties file has

plugin.path=E:/Tools/kafka_2.12-2.4.0/plugins

My MongoSinkConnector.properties file has

name=mongo-sink
topics=first_topic
connector.class=com.mongodb.kafka.connect.MongoSinkConnector
tasks.max=1
key.ignore=true

# Specific global MongoDB Sink Connector configuration
connection.uri=mongodb://localhost:27017,mongo1:27017,mongo2:27017,mongo3:27017
database=test_kafka
collection=transactions
max.num.retries=3
retries.defer.timeout=5000
type.name=kafka-connect

In the E:/Tools/kafka_2.12-2.4.0/plugins folder I have mongo-kafka-connect-1.0.1.jar file.

Command

bin\windows\connect-standalone config\connect-standalone.properties config\MongoSinkConnector.properties

The error I get is

[2020-03-23 04:04:12,376] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone)
java.lang.NoClassDefFoundError: com/mongodb/ConnectionString
        at com.mongodb.kafka.connect.sink.MongoSinkConfig.createConfigDef(MongoSinkConfig.java:140)
        at com.mongodb.kafka.connect.sink.MongoSinkConfig.<clinit>(MongoSinkConfig.java:78)
        at com.mongodb.kafka.connect.MongoSinkConnector.config(MongoSinkConnector.java:62)
        at org.apache.kafka.connect.connector.Connector.validate(Connector.java:129)
        at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:313)
        at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:194)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)
Caused by: java.lang.ClassNotFoundException: com.mongodb.ConnectionString
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

Which other jar files should I place in the plugins folder and/or config changes do I have to make?

UPDATE 1 I have placed mongodb-driver-core-4.0.1 and bson-4.0.1 jar files also in the plugins folder, but have the same error.

Upvotes: 3

Views: 2196

Answers (1)

user1575148
user1575148

Reputation: 589

Finally, I could make the mongo-kafka-connector work on Windows.

Here is what worked for me: Kafka installation folder is E:\Tools\kafka_2.12-2.4.0

E:\Tools\kafka_2.12-2.4.0\plugins has mongo-kafka-1.0.1-all.jar file.

I downloaded this from https://www.confluent.io/hub/mongodb/kafka-connect-mongodb Click on the blue Download button at the left to get mongodb-kafka-connect-mongodb-1.0.1.zip file.

There is also the file MongoSinkConnector.properties in the etc folder inside the zip file. Move it to kafka_installation_folder\plugins

My connect-standalone.properties file has the following entries:

bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=false
value.converter.schemas.enable=false

offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=E:/Tools/kafka_2.12-2.4.0/plugins/mongo-kafka-1.0.1-all.jar

My MongoSinkConnector.properties file has the following entries

name=mongo-sink
topics=topic1,topic2
connector.class=com.mongodb.kafka.connect.MongoSinkConnector
tasks.max=1

connection.uri=mongodb://localhost:27017,localhost:27017,localhost:27017
database=test_kafka
collection=transactions
max.num.retries=3
retries.defer.timeout=5000

field.renamer.mapping=[]
field.renamer.regex=[]

max.batch.size = 0
rate.limiting.timeout=0
rate.limiting.every.n=0

How To Run

Start mongodb, zookeeper, kafka server in three consoles.

In 4th console, start Kafka connect --

bin\windows\connect-standalone config\connect-standalone.properties config\MongoSinkConnector.properties

In 5th console, send msgs to a topic (I did for topic1)

bin\windows\kafka-console-producer --broker-list localhost:9092 --topic topic1

>{"Hello":1}

>{"Mongo":2}

>{"World":3}

Open a mongo client and check your database/collections. You will see these three messages.

Upvotes: 3

Related Questions