CODEWITHSUNDEEP
apache-kafkamessage-queue
Sandeep
Sandeep

Reputation: 3676

Kafka - move messages to front of queue based on scheduled time

Is there a feature in Kafka (or other message queues) through which we can add messages which will become active only of the time specified. Which means that message should not be consumed before the specified time.

Follow up, is there a way through which we can move those messages to the front of queue when it reaches scheduled time.

Upvotes: 0

Views: 2206

Answers (2)

Michal Borowiecki
Michal Borowiecki

Reputation: 4314

Not in the Kafka broker itself, but you can do this using Kafka Streams by defining a custom Processor, which does the following:

  • store each message into a StateStore using the timestamp of when the message should be processed as the key. Since multiple messages may have the same such timestamp, you'll need to wrap them into a collection or suffix the key with some unique value.
  • in the init method schedule a punctuation which periodically does a ranged query on the state store and forwards the messages that have reached their timestamp and removes them from the store.
  • the forwarded messages will go into a separate topic and the consumers will have to consume from that topic instead but the messages there will be in the order you defined.

Upvotes: 1

Jakub
Jakub

Reputation: 3981

Kafka cannot do this. Kafka is basically just a commit log which will write the messages in the order in which they are received and read them in he same order.

However other more traditional messaging brokers can do it - for example ActiveMQ Artemis has support for scheduled messages. I do not think the message will move within the queue, but it will not be consumed before the scheduled time.

Upvotes: 1

Related Questions