CODEWITHSUNDEEP
mongodbapache-kafkaapache-kafka-streams
user3276247
user3276247

Reputation: 1084

Change state store for Kakfa Streams to use MongoDB

Default state store is rocksdb. I could also find in-memory state stores. However, I need to figure out if kafka streams can be configured to use mongodb as the state store and also if I can define those collections myself.

Upvotes: 2

Views: 795

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191728

You'd implement a StateStore and StsteStoreSupplier, however, it's recommended that you not use remote data stores as this will cause inconsistencies and considerable lag in the topology that have already been taken into account for the RocksDB implementation. It'll also cause high churn of many reads and writes on the database, which will affect all clients

What you'd find commonly implemented instead is using Kafka Connect MongoDB sinks to write data after being fully processed using built-in state store options and Debezium or Mongo source connectors to get data into Kafka for processing

You may find a POC for a Redis statestore here, though, as an inspiration for trying to write your own for Mongo - https://github.com/andreas-schroeder/redisks

Upvotes: 1

Related Questions