CODEWITHSUNDEEP
postgresqlapache-kafkadebezium
styx
styx

Reputation: 21

Debezium stops after initial sync

The initial sync works as expected but then the connector just stops and does not care about further table changes. There are no errors thrown and the connector is still marked as active and running.

Database: Amazon Postgres v10.7

Debezium config:

  "name": "postgres_cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "...",
    "database.port": "5432",
    "database.user": "...",
    "database.password": "...",
    "database.dbname": "...",
    "database.server.name": "...",
    "table.whitelist": "public.table1,public.table2,public.table3",
    "plugin.name": "pgoutput",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "unwrap, route, extractId",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": false,
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "[^.]+\\.[^.]+\\.(.+)",
    "transforms.route.replacement": "postgres_$1",
    "transforms.extractId.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.extractId.field": "id"
  }
}

Any thoughts about what the problem could be?

Edit:

Log-Errors:

ERROR WorkerSourceTask{id=postgres_cdc-0} Failed to flush, timed out while waiting for producer to flush outstanding 75687 messages (org.apache.kafka.connect.runtime.WorkerSourceTask)
ERROR WorkerSourceTask{id=postgres_cdc-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter)

Upvotes: 1

Views: 742

Answers (1)

Nevada Perry
Nevada Perry

Reputation: 11

You can diagnose whether it's a Postgres issue or a connector issue by running these four commands on Postgres, assuming logical replication is enabled:

select * from pg_create_logical_replication_slot('test', 'test_decoding', false, false);

create table planet (name text);

insert into planet (name) values ('earth');

select * from pg_logical_slot_peek_changes('test', null, null);

If it comes back with transaction data:

    lsn    | xid  |                      data
-----------+------+-------------------------------------------------
 0/4021CD0 | 8734 | BEGIN 8734
 0/4021CD0 | 8734 | table public.planet: INSERT: name[text]:'earth'
 0/4021D40 | 8734 | COMMIT 8734
(3 rows)

then Postgres is fine and the issue is likely with Debezium.

If it comes back with no data:

 lsn | xid | data
-----+-----+------
(0 rows)

then you've isolated it as a Postgres issue.

In our case, we isolated the issue as a bug/discrepancy that happens only on:

  1. Amazon Aurora PostgreSQL
  2. when synchronous_commit = off.

but not on any standard PostgreSQL instance with synchronous_commit = off. We filed a bug report with AWS regarding this issue.

To clean up:

select pg_drop_replication_slot('test');

Regarding the log message Failed to flush / Failed to commit offsets, I couldn't find any definitive answer on whether the data is intact, though this author suspects that the data is fine and only performance is affected.

Upvotes: 1

Related Questions