KafkaIO checkpoint persistence with Google Dataflow Runner

Question

I am trying to understand how the offsets and group management works with the Google Dataflow runner with KafkaIO reader. More specifically, I am trying to understand how offset management works:

• If the group.id config is set and if auto-commit and commitOffsetsInFinalize are disabled.
• If the group.id config is not set, how does offset and group management work?

Any code/document reference pointing in the right direction is appreciated.

Answer 1

The KafkaIO reader is entirely part of Apache Beam. Google Cloud Dataflow does not treat this source differently than any other Beam source.

You can find its code at https://github.com/apache/beam/tree/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka (in various files). I don't know of any reference documentation other than the Javadoc.