CODEWITHSUNDEEP
mysqlpostgresqlapache-kafka-connect
Ryuzaki L
Ryuzaki L

Reputation: 40078

How kafka connector works for postgresql and mysql database

I'm following the quick start tutorials from here quick-start-kafka-connect

This tutorial shows how to stream the mysql database table changes into kafka topic.

The only part is download everything and just add /tmp/kafka-connect-jdbc-source.json file with some config properties and start

How does this work in background ?

1 : Does it create connection with database and monitor tables for specific intervals of time? OR

2 : Does it uses replication log? (i don't know how this works)

3 : Is this same mechanism for mysql and postgresql?

Upvotes: 0

Views: 7438

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191884

Debezium monitors the OpLog.

Kafka Connect JDBC by Confluent (which you've linked to) can use a time-interval, and that configuration is shared by all JDBC-compliant connections, MySQL and Postgres included.

For incremental query modes that use timestamps, the source connector uses a configuration timestamp.delay.interval.ms ...

replication log? (i don't know how this works)

You can find the Debezium guide here, but this mechanism differs for Mongo, Postgres, MySQL, MSSQL, Oracle, etc.

Upvotes: 2

Related Questions