CODEWITHSUNDEEP
apache-kafkardbmsapache-kafka-connect
Evaldas Raisutis
Evaldas Raisutis

Reputation: 1638

How does Kafka know when source data has changed?

I can't find a definitive answer, so I figured I would ask the experts. How does Kafka observe and detect what data in a given source has changed? For instance, in a Relational Database?

Polling comes to mind, but wouldn't it then have to maintain a data set of all primary keys per available table, and then run checks if new primary keys are available? Where is this stored, since memory is probably not durable enough?

Upvotes: 0

Views: 549

Answers (1)

dawsaw
dawsaw

Reputation: 2313

This is a very general question so you can imagine the answer is "it depends". Kafka isn't tracking this per se. It's done by whatever Kafka client implementation you have. For example, if you implement a Kafka Connect source connector then you can store offsets to checkpoint what data has been read in Kafka itself. If you are just writing a producer it's a different story. A pretty general example can be found in the Confluent JDBC source connector. It has multiple modes for loading that can give you an idea of the flexibility https://docs.confluent.io/current/connect/connect-jdbc/docs/source_connector.html#features

Upvotes: 1

Related Questions