CODEWITHSUNDEEP
javaapache-kafka
Alex Ivv
Alex Ivv

Reputation: 3

How to dinamically apply scheduled kafka consumer based on topic?

I'm currently struggling with a consumer on kafka that can somehow schedule to a future time for execution.

To summarize: I have a big data storage (.csv file) and the records contains 2 columns: timestamp and value. I'm trying to process this values based on their timestamp. First record it has to be consumed instantly by kafka, next one should be processed in future with a delay of 'current record timestamp - previous record timestamp' (it is not a very big difference, just a few seconds = result will be in millis) and so on.

So basically I can't find a solution to implement a consumer on kafka that takes each records based on timestamp and use that exact delay. I have to just simulate these values and they have to be insert in DB accordly to that delay to work properly.

I've tried to work around threads, with executors, but for big data it's not a properly way.

I tried to create dynamic topics on producers based on timestamp and then subscribe to them and then somehow process with a queue. It didn't work.

I expect the kafka to consume each record with the delay based on timestamp.

Upvotes: 0

Views: 1673

Answers (1)

ADS
ADS

Reputation: 718

I expect the kafka to consume each record with the delay based on timestamp

If you have specific delay between messages then Kafka is not a proper solution. When you send messages to the Kafka, in most scenarios you use network. Which could add its own, unpredictable, delay. Kafka is running as a different process and nobody could guarantee at which moment this process will be ready to receive next message. OS could suspend process, GC could start etc. This adds another delay which nobody could predict.
Also, Kafka is not designed to operate with time when message was received. It more focused on order of messages, low latency and high throughput but not on timing.

Upvotes: 1

Related Questions