CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
JavaTechnical
JavaTechnical

Reputation: 9347

Can we have sequential processing of messages with Kafka Streams API?

We have some messages which we need to maintain sequence for. We have decided to send all messages from a particular source to a partition, so that sequence of messages will be maintained (multiple sources can produce to the same partition but a source cannot produce to multiple partitions) and we will be able to identify each source with their key.

Now, we need to consume those messages and do some processing. We do multiple independent operations on the messages what we have consumed (for example, storing them in the database, forwarding them etc). Now, I am stuck with whether to use the Kafka Streams API or Consumer API for this.

Note: I cannot have large number of topics (for example, I cannot create a topic for each source as the sources will be numerous). Though I can group messages by key to identify the source, for using Streams, what I want is the order of messages for a key.

Usecase: I want to commit those messages into a database in order and I want to forward those messages in order.

So how can I process the messages in order using Streams API?

Upvotes: 3

Views: 4210

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191701

We have decided to send all messages from a particular source to a partition

Kafka guarantees order within a single partition, by the order they arrived at the broker, not by any other field or by time. All Kafka clients (Consumer, Streams, third-party libraries) respect this fact.

Generally, though, if you included the message timestamp as part of your database insert event, then it would be possible to group by key, order by timestamp. That will depend on the database, though.

If you're already partitioning by a known source key, why don't you just filter by that in the streams app? Otherwise, you will have to use Consumer API because it allows you to assign specific partitions (the Processor API, might, but haven't used it)

Upvotes: 4

Related Questions