CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-apispring-kafka
tzortzik
tzortzik

Reputation: 5143

spring kafka looking for the latest available message in a topic

I have a Kafka consumer which subscribes to the following topics MY_TOPIC, MY_UNINTERESTED_TOPIC.

In the following scenario I am not interested in the second topic but I had to mention it because if I configure it using something like auto.offset.reset it may affect all the topics.

On MY_TOPIC topic I publish different kind of messages: MESSAGE_TYPE_A and MESSAGE_TYPE_B. Both messages are instances of BaseKafkaMessage (custom class) with different properties.

Now I am interested in finding only the latest message of type MESSAGE_TYPE_A. How can I do that?

The real scenario is this one: I publish two types of messages on the same topic. One of them is used to prepare a local cache in each consumer that is interested in this topic and that message. If a consumer stops, when it reloads it must reinitialize its cache with the latest MESSAGE_TYPE_A. MESSAGE_TYPE_B should be ignored. I don't want to send a notification on Kafka to the data provider to publish the data again because all the subscribers will have a lot of unnecessary work to do.

How can I obtain this? Is this possible?

I found https://docs.spring.io/spring-kafka/reference/htmlsingle/#seek but I am not sure if it is what I am looking for or if there is another way to do it.

Upvotes: 0

Views: 725

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 192013

It's not clear what format these messages are in or why you really need them to be in the same topic at all.

You could use different Avro types for example. Or you would have to try-catch parsing two different byte arrays (JSON objects?)

Or you could partition your topic by type such that all messages of one type are in-order of the same partition.

But, there is no mechanism to do index-lookups to get the most-recently sent message. Either you start from the latest event, and you get the next incoming message, or you can start from the beginning, then scan up until you get 0 records on the next poll loop, which in-theory is the "most-recent"

something like auto.offset.reset it may affect all the topics

That only affects those topics for which the consumer is interested it, not all.

Upvotes: 3

Related Questions