How to connecting kafka to mysql?

Question

I am Trying to connect kafka with mysql on windows. I am not using confluent. My kafka version is 2.12 I have started zookeeper, Kafka, producer and consumer, This all works fine.

My MysQL version is 8.0.15

And I have copied this 3 jar files in the libs folder

mysql-connector-java-8.0.15.jar
mysql-connector-java-5.1.47.jar
mysql-connector-java-5.1.47-bin.jar

My source-quickstart-mysql.properties file code is

name=test-source-mysql-jdbc-autoincrement        connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://localhost:3306/databasename? 
user=rootname&password=password
mode=incrementing
incrementing.column.name=ID
topic.prefix=my-replicated-topic-table1

When I run the command

connect-standalone.bat ..\..\config\connect-standalone.properties  ..\..\config\source-quickstart-mysql.properties

I am getting this Error on console

[2019-03-26 16:16:39,524] ERROR Failed to create job for ....\config\source-quickstart-mysql.properties (org.apache.kafka.connect.cli.ConnectStandalone) [2019-03-26 16:16:39,524] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone) java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourc eConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.1.0', encod edVersion=2.1.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSource Connector', version='2.1.0', encodedVersion=2.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.co nnect.tools.MockConnector', version='2.1.0', encodedVersion=2.1.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.1.0', encodedVersion=2.1.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tool s.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.1.0', encodedVersion=2.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class o rg.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.1.0', encodedVersion=2.1.0, type=source, typeName='source', location='class path'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.1.0', encodedVersion=2.1.0, type=source , typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2 .1.0', encodedVersion=2.1.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79) at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:110) Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector, available connecto rs are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.1.0', encodedVersion=2.1.0, type=sink, ty peName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.1.0', e ncodedVersion=2.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', ve rsion='2.1.0', encodedVersion=2.1.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connec t.tools.MockSinkConnector', version='2.1.0', encodedVersion=2.1.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='o rg.apache.kafka.connect.tools.MockSourceConnector', version='2.1.0', encodedVersion=2.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools. SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.1.0', encodedVersion=2.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.1.0', encodedVersion=2.1.0, type=source, typeName='source', location= 'classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.1.0', encodedVersion=2.1.0, t ype=source, typeName='source', location='classpath'} at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:179) at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:382) at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:261) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:189) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)

Please help me.

I have also tried this article but no output generated with command bin/confluent load jdbc-source -d jdbc-source.properties https://supergloo.com/kafka-connect/kafka-connect-mysql-example/

Answer 1

You could download Kafka Connect JDBC at https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc which is free and works without Confluent Platform. After unzipping, update the location of confluentinc-kafka-connect-jdbc-5.5.1.jar for the key plugin.path in connect-standalone.properties available in the config directory of installed Apache Kafka. The error will go off after running the script again.

Answer 2

if you add kafka-connect-jdbc-5.5.1.jar in kafka lib path and restart kafka zookeeper and server. You should able to connect.

Answer 3

Your error is

 org.apache.kafka.connect.errors.ConnectException: Failed to find any class that 
 implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector

Which since you say you're not using Confluent Platform makes sense, because kafka-connect-jdbc is not part of Apache Kafka. You can either use Confluent Platform, build the connector from source, or download it at http://hub.confluent.io.