CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectdebeziumstrimzi
MiH
MiH

Reputation: 421

KafkaConnect fails to load plugin

I'm deploying KafkaCluster, KafkaConnect on K8s by using Strimzi.

I'm trying to setting KafkaConnector which connects to Postgres DB.

These are my steps:

  1. Build a new KafkaConnect from this image 'quay.io/strimzi/kafka:0.26.1-kafka-3.0.0'. This new image has plugin which I download from this link 'https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-deployment'

  2. I run this new image on K8s, then exec into KafkaConnect container. It has plugin correctly ('/opt/kafka/plugins/debezium/debezium-connector-postgres'): enter image description here

  3. I create KafkaConnector which refers to class 'io.debezium.connector.postgresql.PostgresConnector':

kind: "KafkaConnector"
metadata:
  name: dev-approvalflowservice-connector
  labels:
    strimzi.io/cluster: postgres-connect-cluster-1
spec:
  class: io.debezium.connector.postgresql.PostgresConnector
  tasksMax: 2
  config:
    database.hostname: 10.14.101.204
    database.port: "5432"
    database.user: "debezium"
    database.password: "debezium"
    database.dbname: "approvalflowservice"
    database.server.name: "approvalflowservice"
    plugin.name: "pgoutput"
    slot.name: "approvalflowserviceslot"
    truncate.handling.mode: "include"
    table.include.list: "public.approval_flow"
  1. The problem is KafkaConnect fail to load the Plugin which I already added in to new KafkaConnect image. Log from KafkaConnect: log

Update

  1. I tried to change the location of plugin inside the new KafkaConnect image ('/opt/kafka/plugins/debezium-connector-postgres') : enter image description here
  2. The problem is the same: log

Thank you all.

Upvotes: 0

Views: 1322

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

In your logs, you have

Loading plugin from: /opt/kafka/plugins/debezium

But, since your plugin is installed into a subfolder, no Postgres class is being listed as loaded.

Make sure that the plugin path only has one folder between the path you've defined and where the JARs are available. e.g. install to /opt/kafka/plugins/debezium-connector-postgres

You can also issue a GET /connector-plugins HTTP request to the Connect API to list the available connectors for the cluster before you create one.

Upvotes: 0

Related Questions