AlessioG
AlessioG

Reputation: 614

Kafka Connect: streaming changes from Postgres to topics using debezium

I'm pretty new to Kafka and Kafka Connect world. I am trying to implement CDC using Kafka (on MSK), Kafka Connect (using the Debezium connector for PostgreSQL) and an RDS Postgres instance. Kafka Connect runs in a K8 pod in our cluster deployed in AWS.

Before diving into the details of the configuration used, I'll try to summarise the problem:

My connector config looks like:

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.user": "root",
    "database.dbname": "insights",
    "slot.name": "cdc_organization",
    "tasks.max": "1",
    "column.blacklist": "password, access_key, reset_token",
    "database.server.name": "insights",
    "database.port": "5432",
    "plugin.name": "wal2json_rds_streaming",
    "schema.whitelist": "public",
    "table.whitelist": "public.kafka_connect_cdc_test",
    "key.converter.schemas.enable": "false",
    "database.hostname": "de-test-sre-12373.cbplqnioxomr.eu-west-1.rds.amazonaws.com",
    "database.password": "MYSECRETPWD",
    "value.converter.schemas.enable": "false",
    "name": "source-postgres",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "snapshot.mode": "initial"
}

We have tried different configurations for the plugin.name property: wal2josn, wal2json_streaming and wal2json_rds_streaming.

There's no problem of connection between the connector and the DB as we already saw messages flowing through as soon as the connector starts.

Is there a configuration issue with the connector described above that prevent us to see messages related to new changes appearing in the topic?

Thanks

Upvotes: 0

Views: 1135

Answers (0)

Related Questions