CODEWITHSUNDEEP
apache-kafka
Sheena
Sheena

Reputation: 117

Apache Kafka: Send Messages to another Topic after a period of time

I am new to Apache Kafka, so it might be that this is basic knowledge.
At the moment I try to figure out some possibilities and functions that Kafka offers me. And so I was wondering whether it is possible to move a message after a specified period of time to another topic.

Scenario:
Producer 1 writes Message (M1) into Topic 1 where Consumer 1 handles the messages.
After a period of time, let's say 1 hour, M1 is moved into Topic 2 to which the Consumer 2 is subscribed.

It is possible to do something like that with Kafka? I know that there is a way to delete a message after a period of time, but I don't know if there is a way to change to topic or catch the delete-action.

I thought about running a timer in a Producer, but with a huge amount of data, I think that this isn't possible anymore.

Thanks in advance

EDIT:
Thanks to @OneCricketeer i know, that my first assumption with the several producers wasn't that bad. I know that the throughput with one Producer is really good and that one won't take the system down. But I'm still concerend about the second producer.
In my imagination it is like the following sketchy image
Sketchy example of the process

When I take 30 messages per minute that would mean that I would habe 31 instances of producers. 1 that handles the messages asap and 30 others waiting for the timer to determinate so that they can work with their message.
Counting that up to an hour it would be round about 1800 instances. That is where I#m concerned about. Or is there a better way to handel this?

Upvotes: 1

Views: 2282

Answers (1)

Sheena
Sheena

Reputation: 117

I found a solution that might work for my case. I accidentally stumbled over a Consumer-Methode which allows you to read messages based on Timestamp. The methode is called offsetsForTimes and usable since the Version 0.10.

See the Kafka API or the following post which I found researching about that methode.

Maybe this is usefull for others so I decided to publish this.

Upvotes: 1

Related Questions