CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Vinod Jayachandran
Vinod Jayachandran

Reputation: 3898

Kafka Jdbc Connect timestamp+incrementing mode

I am using Kafka Jdbc Connect timestamp+incrementing mode to sync a table rows to Kafka. Reference https://docs.confluent.io/current/connect/connect-jdbc/docs/source_config_options.html#mode

The challenge is the table gets synced from the beginning of time since the start time by default is 1970. Is there any way to over ride the start time (i.e) I want to sync only from the beginning of input given date.

Upvotes: 2

Views: 1572

Answers (2)

kaminzo
kaminzo

Reputation: 336

You need to set the timestamp.initial to the input given date you desire. It needs to be set in the epoch format.

The SQL query in timestamp+incrementing mode is appended with:

WHERE "DBTime" < 'system high date i.e. 9999-12-12T23:59:59+00:00' AND (("DBTime" = 'timestamp.initial' AND "DBKey" > '-1') OR "DBTime" > 'timestamp.initial') ORDER BY "DBTime","DBKey" ASC

https://docs.confluent.io/kafka-connect-jdbc/current/source-connector/source_config_options.html#mode

Upvotes: 2

marius_neo
marius_neo

Reputation: 1595

In case that you want to start from a given offset with your connector, I'd suggest overwriting the information stored in the connect-offsets topic.

Through the Kafka REST API you can easily read the content of this topic:

http://localhost:8082/topics/connect-offsets

Looking through the code of the kafka-connector-jdbc in the relevant methods for the use case that you've described:

  • io.confluent.connect.jdbc.source.TimestampIncrementingCriteria#extractValues
  • io.confluent.connect.jdbc.source.TimestampIncrementingOffset#getTimestampOffset

that overriding the connect-offsets topic content seems to be the only way available at the moment.

Upvotes: 0

Related Questions