CODEWITHSUNDEEP
jdbcapache-kafkaapache-kafka-connect
Debbie R
Debbie R

Reputation: 53

Kafka topic order

I have a Kafka Connector using something like the following:

SELECT CAST ( id as VARCHAR) as key_id, id ...
FROM table1 
JOIN (SELECT id as tID FROM table1 t WHERE t.id = 87002) v 
ON v.tID = id

If I execute against the DB, with ORDER BY ID I get the records in the order I expect with the following Ids

322633
324066
324084
324107

I have specified the connector pull incrementing, so the following is tacked on by the connector:

WHERE id > ? ORDER BY id ASC

Here's where I am confused. In the stream this connector writes to, the records show up in a different order! They are in 

322633
324084
324107
324066   (huh!!!?)

Any help understanding this behavior would be great.

Upvotes: 2

Views: 284

Answers (1)

Nitin
Nitin

Reputation: 3842

Kafka is a distributed messaging system hence even its fetch message from source DB in sequential order but the message will get distributed on multiple partitions of the topic based on the defined key. Kafka guarantee to maintain order per partition not across the partition.

So consumers can get consume messages from the topic in any order there is no guaranteed order on the consumer side.

If you looking strictly to maintain order, you can use a single partition topic but in that case, you can lose parallelism.

enter image description here

Upvotes: 1

Related Questions