CODEWITHSUNDEEP
postgresqlapache-kafka-connectdebeziumwal
Suren Aznauryan
Suren Aznauryan

Reputation: 1094

Debezium `source.lsn` field value in snapshot records(when `__op=r`)

I can't find an information in Debezium Postgresql connector config documentation page regarding the source.lsn field value for snapshot events(events having __op=r). I've ran a test on my dev environment database, which doesn't have much load, and I have got non-null lsn value for all events having __op=r. As my db doesn't have much load I assume that all WAL segments created from the beginning are still present, though not sure about that.

My questions are the following:

  1. How does debezium get a non-null lsn value for snapshot records if it just gets that records' info by issuing select queries on all existing data(in contrast of checking the WAL for generating events having __op=c or __op=u or __op=d) ? Is the WAL anyway checked during snapshotting ?
  2. Assuming that the WAL is checked during snapshotting, what will be the value of lsn in snapshot event in case if the corresponding WAL segment is already deleted ? Is lsn equal to null in such case ?
  3. Is there any documentation from Debezium describing the possible lsn values in case of snapshotting ?

Upvotes: 0

Views: 1711

Answers (1)

Naros
Naros

Reputation: 21133

How does debezium get a non-null lsn

The source.lsn value for a new snapshot is sourced from the consistent_point that available via the newly created replication slot when the connector began. This makes sense as its the point when the slot was created and the timepoint in which the exported snapshot is taken.

what will be the value of lsn in snapshot event in case if the corresponding WAL segment is already deleted ? Is lsn equal to null in such case ?

When a replication slot is created and there are entries in the WAL that are pending delivery to the consumer of the replication slot, WAL segments cannot be freed until PG says they're no longer needed. AFAIK, you cannot achieve the state you described.

Is there any documentation from Debezium describing the possible lsn values in case of snapshotting ?

We don't cover this in the documentation but it should be possible to see by querying the replication slot while a snapshot is happening and seeing the starting LSN value is identical to that of the records in the snapshot.

Upvotes: 0

Related Questions