Reputation: 11
I'm new to debezium CDC and I tested using debezium-server to stream data from mysql 5.5 to pubsub sink. I know the version of mysql is very old and not officially supported based on debezium documentation but I need to use this version.
I deployed debezium-server on local machine with Docker Compose as below. This works perfectly find. Debezium-server version is 2.6.0.Final, I can see this driver file in /lib/mysql-connector-j-8.2.0.jar.
# docker-compose.yaml
services:
debezium-server:
image: quay.io/debezium/server:2.6.0.Final
container_name: debezium-server
ports:
- 8080:8080
environment:
- PUBSUB_EMULATOR_HOST=pubsub:8085
- PUBSUB_PROJECT_ID=abc
volumes:
- $PWD/debezium-server/conf:/debezium/conf
- $PWD/debezium-server/data:/debezium/data
- $PWD/debezium-server/lib/mysql-connector-java-5.1.40-bin.jar:/debezium/lib/mysql-connector-java-5.1.40-bin.jar # try this which gives "class not found" error
depends_on:
- postgres
- pubsub
pubsub:
container_name: pubsub
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:414.0.0-emulators
ports:
- 8085:8085
command:
- gcloud
- beta
- emulators
- pubsub
- start
- --project=abc
- --host-port=0.0.0.0:8085
# application.properties
# Source Configuration --
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory
debezium.source.schema.history.internal.file.filename=data/schemahistory.dat
debezium.source.database.hostname=${HOST}
debezium.source.database.port=3306
debezium.source.database.user=${USER}
debezium.source.database.password=${PASSWORD}
debezium.source.database.dbname=${DB}
debezium.source.database.server.id=1
debezium.source.tombstones.on.delete=false
# Sink Configuration --
debezium.sink.type=pubsub
debezium.sink.pubsub.project.id=abc
debezium.sink.pubsub.address=pubsub:8085
#Quarkus Configuration --
quarkus.log.console.json=false
Now when I tried streaming with mysql 5.1, debezium breaks with CLIENT_PLUGIN_AUTH error. I assumed this is because the driver is not compatible with mysql 5.1. When I tried mounting mysql-connector-driver-5.1.40-bin.jar to /lib/, debezium breaks with "java class not found" error. Is there a way to use older version of mysql driver to make debezium-server compatible with mysql 5.1? Thank you.
Upvotes: 0
Views: 39