CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-api
kan
kan

Reputation: 28981

Kafka consumer offset auto-reset and other parameters

I have a KafkaConsumer which needs to subscribe to two topics topicA and topicB. However I need some parameters be different. E.g. if I need auto.offset.reset for topicA be earilest while for topicB it should be latest. I see no easy way to do that. One option is to run two consumers, but in this case I need two polling threads for them and hence should handle multi-threading. Is there any simpler way?

Upvotes: 0

Views: 424

Answers (2)

sawim
sawim

Reputation: 1082

If you need to use one consumer, then I think you could set auto.reset.offset to latest and then move offsets of topicA manually (if needed). To do so, between subscription and poll loop you could:

  1. Get partitions assigned to the consumer (method assignment)
  2. Check commited offsets of topic topicA by filtering partitions of topicA from the previous point (method committed). If the result is null, then call seekToBeginning method.

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 191983

Creating two (or more) threads is correct.

Consumers are not thread-safe, and should be isolated and separated from other processes anyway.

You could use higher-level Kafka libraries (e.g. Vert.x / Spring) for simplifying this.

Upvotes: 1

Related Questions