CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
HISI
HISI

Reputation: 4797

Stateful and Stateless consumer on Kafka

On Kafka a consumer could be stateful or stateless, what is the different between these two types of consumers ? why the stateful consumer should rebuild the state when assigned new partitions?

Upvotes: 5

Views: 4120

Answers (1)

Michael Heil
Michael Heil

Reputation: 18495

A KafkaConsumer itself is just polling data from a Kafka topic and it can't be stateful nor stateless. However, you can have a stateful or a stateless application. Some simple examples would be

  • stateless: consume message and put it into an external storage
  • stateful: consume messages, define a window (=time interval) and calculate the number of messages per any kind of identifier

In a stateful (KafkaStreams) applications the actual state is stored in an internal state store such as RocksDB. If a rebalance happends (or a new partitions gets assigned) this internal state store has to be build up first to be "up-to-date" with the state before actually being stateful again.

There are plenty of good materials on the web, like here or here

Upvotes: 2

Related Questions