CODEWITHSUNDEEP
apache-kafkakafka-rest
Ravi P
Ravi P

Reputation: 1

How to consume data from Kafka topic from specific offset using REST Proxy for Kafka?

I am using Kafka Rest Proxy to produce as well as consume from Kafka topic using Azure APIM, hence the reason to use Rest Proxy. I can consume messages in a linear fashion using auto.offset.reset = earliest, so every call to Kafka Rest will give me the latest message, which is great However,

In below scenario:

A1 is data requestor
B2 is API that fulfills A1's request using REST proxy for Kafka
C3 is Kafka Rest proxy server

If A1 requests data from B2, and B2 consumes message from kafka topic C3 to give to A1, but due to some internet issue that message could not make it to A1. So now A1 will have to make another request to get the lost message, BUT the offset is already incremented in Kafka Rest Proxy. So now if B2 consumes again it will be new data not the data that was lost. How can I use offset in Kafka Rest Proxy to read older messages?

Upvotes: 0

Views: 664

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191884

auto.offset.reset = earliest only applies when a group does not yet exist. It doesn't affect any following requests. It is also the start of the topic, not the "latest".

You need to commit offsets for a group. E.g.

POST /consumers/(string:group_name)/instances/(string:instance)/offsets
Content-Type: application/vnd.kafka.v2+json

{
  "offsets": [
    {
      "topic": "test",
      "partition": 0,
      "offset": 20
    },
    {
      "topic": "test",
      "partition": 1,
      "offset": 30
    }
  ]
}

Then the next consumer record batch will start consuming from those offsets.

Upvotes: 0

Related Questions