GauravRatnawat
GauravRatnawat

Reputation: 859

Kafka go-lang working of enable.auto.offset.store=false with auto.commit = false

Using go-lang 1.18 with confluent-kafka-go v1.8.2

used enable.auto.commit = false config. we are manually committing the offset once the successful processing of the message. However, even after setting this config once we got an error while processing, we are not seeing the message with the same key. (that means somehow offset is getting committed even in error scenarios ). Note: for the error scenario, it took 8-9 sec to process and completely call it an error.

Also got this from link It is recommended to set `enable.auto.offset.store=false` for long-time processing applications and then explicitly store offsets (using offsets_store()) after message processing, to make sure offsets are not auto-committed prior to processing has finished.

Que:

  1. By default how much time does kafka wait till it auto-commits the offset.
  2. Do we have a mechanism to stop this auto-commit at all.

Upvotes: 1

Views: 2315

Answers (2)

David Connell
David Connell

Reputation: 1

Committing an offset is basically moving the stack pointer and committing all records up to that offset. So, as soon as you successfully process and commit an offset (which should be offset+1 btw), you are committing all the error records too. Manual commit also has to deal with the rebalancing act which can also be a challenge. I find it easier and more accurate to set auto commit and then repost things to the topic that would qualify for a retry. But that may be because I'm just being lazy lol.

Upvotes: 0

Gerard Garcia
Gerard Garcia

Reputation: 1856

Offsets are committed at the intervals configured in auto.commit.interval.ms, which by default is 5 seconds.

Setting enable.auto.commit to false should be enough to disable auto-committing completely.

Upvotes: 0

Related Questions