debezium postgres cant find `io/debezium/util/IoUtil`

Question

I have a problem trying to run kafka connect with postgres debezium connector while using docker, it cant find a certain class that is located in a debezium-core jar.

I have downloaded the debezium postgres plugin and copied its content to the plugin.path path that I have defined (/kafka/plugins).

Even if kafka connect loads first debezium core and api, the error still shows up.

This is the log:

kafka-connect_1  | [2020-08-13 14:06:13,543] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed)
kafka-connect_1  | [2020-08-13 14:06:13,564] INFO Loading plugin from: /kafka/plugins/debezium-api-1.3.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,628] INFO Registered loader: PluginClassLoader{pluginLocation=file:/kafka/plugins/debezium-api-1.3.0.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,628] INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,628] INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,629] INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,629] INFO Loading plugin from: /kafka/plugins/debezium-core-1.3.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,762] INFO Registered loader: PluginClassLoader{pluginLocation=file:/kafka/plugins/debezium-core-1.3.0.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,762] INFO Added plugin 'io.debezium.converters.ByteBufferConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,762] INFO Added plugin 'io.debezium.converters.CloudEventsConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,763] INFO Added plugin 'io.debezium.transforms.outbox.EventRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,763] INFO Added plugin 'io.debezium.transforms.ExtractNewRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,763] INFO Added plugin 'io.debezium.transforms.ContentBasedRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,763] INFO Added plugin 'io.debezium.transforms.ByLogicalTableRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,763] INFO Added plugin 'io.debezium.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,763] INFO Loading plugin from: /kafka/plugins/protobuf-java-3.8.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,879] INFO Registered loader: PluginClassLoader{pluginLocation=file:/kafka/plugins/protobuf-java-3.8.0.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,879] INFO Loading plugin from: /kafka/plugins/debezium-connector-postgres-1.3.0.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
kafka-connect_1  | [2020-08-13 14:06:13,917] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
kafka-connect_1  | java.lang.NoClassDefFoundError: io/debezium/util/IoUtil
kafka-connect_1  |  at io.debezium.connector.postgresql.Module.<clinit>(Module.java:19)
kafka-connect_1  |  at io.debezium.connector.postgresql.PostgresConnector.version(PostgresConnector.java:47)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:380)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:385)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:355)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:328)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:261)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:253)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:222)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:199)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:60)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
kafka-connect_1  | Caused by: java.lang.ClassNotFoundException: io.debezium.util.IoUtil
kafka-connect_1  |  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
kafka-connect_1  |  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
kafka-connect_1  |  at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
kafka-connect_1  |  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

I have tried the different solutions proposed in other questions that I have found on stackoverflow and other sites, non of the seem to be helpful, reason being is that some of the plugins are recognized and are loaded successfully, but the debezium-connect-postgres one is not...

Any solution to the problem? Thanks!

Answer 1

While following the Deploying Debezium using the new KafkaConnector resource

For the error: java.lang.NoClassDefFoundError: io/debezium/util/IoUtil i performed following steps.

It is assumed that you have already downloaded connector. If not, you can download debezium-connector-mysql from maven repo. I used version 1.9.5.Final

Note: It's better to see version compatibility from Debezium Releases Overview

Once you have extracted the contents of connector, following are the contents of my Dockerfile:

FROM quay.io/strimzi/kafka:0.28.0-kafka-3.1.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/
USER 1001

Having considered we have Dockerfile in current path and extracted folder debezium-connector-mysql

docker build -t debezium-connector-mysql:0.28.0-kafka-3.1.0 .

Answer 2

The solution was to copy the entire folder as is into the container instead of copying each and every file to the plugin path.

In my docker file: Instead of performing COPY ./docker/kafka-connect/debezium-connector-postgres /kafka/plugins,

I've put debezium-connector-postgres folder which contains the plugin into another folder named connectors that is inside ./docker/, and then the action performed changes to COPY ./docker/kafka-connect/connectors /kafka/plugins which results in a new folder inside /kafka/plugins of the container.