How to use a persisted StateStore between two Kafka Streams

Question

I'm having some troubles trying to achieve the following via Kafka Streams:

• At the startup of the app, the (compacted) topic alpha gets loaded into a Key-Value StateStore map
• A Kafka Stream consumes from another topic, uses (.get) the map above and finally produces a new record into topic alpha
• The result is that the in-memory map should aligned with the underlying topic, even if the streamer gets restarted.

My approach is the following:

val builder = new StreamsBuilderS()

val store = Stores.keyValueStoreBuilder(
   Stores.persistentKeyValueStore("store"), kSerde, vSerde)
)

builder.addStateStore(store)

val loaderStreamer = new LoaderStreamer(store).startStream()

[...] // I wait a few seconds until the loading is complete and the stream os running

val map = instance.store("store", QueryableStoreTypes.keyValueStore[K, V]()) // !!!!!!!! ERROR HERE !!!!!!!!

builder
  .stream("another-topic")(Consumed.`with`(kSerde, vSerde))
  .doMyAggregationsAndgetFromTheMapAbove
  .transform(() => new StoreTransformer[K, V]("store"), "store")
  .to("alpha")(Produced.`with`(kSerde, vSerde))

LoaderStreamer(store):

[...]
val builders = new StreamsBuilderS()

builder.addStateStore(store)

builder
  .table("alpha")(Consumed.`with`(kSerde, vSerde))

builder.build
[...]

StoreTransformer:

[...]
override def init(context: ProcessorContext): Unit = {
  this.context = context
  this.store = 
    context.getStateStore(store).asInstanceOf[KeyValueStore[K, V]]
}

override def transform(key: K, value: V): (K, V) = {
  store.put(key, value)
  (key, value)
}
[...]

...but what I get is:

Caused by: org.apache.kafka.streams.errors.InvalidStateStoreException:
The state store, store, may have migrated to another instance.

while trying to get the store handler.

Any idea on how to achieve this?

Thank you!

Answer 1

You can't share state store between two Kafka Streams applications.

According to documentation: https://docs.confluent.io/current/streams/faq.html#interactive-queries there might be two reason of above exception:

• The local KafkaStreams instance is not yet ready and thus its local state stores cannot be queried yet.

• The local KafkaStreams instance is ready, but the particular state store was just migrated to another instance behind the scenes.

The easiest way to deal with it is to wait till state store will be queryable:

public static <T> T waitUntilStoreIsQueryable(final String storeName,
                                              final QueryableStoreType<T> queryableStoreType,
                                              final KafkaStreams streams) throws InterruptedException {
  while (true) {
    try {
      return streams.store(storeName, queryableStoreType);
    } catch (InvalidStateStoreException ignored) {
      // store not yet ready for querying
      Thread.sleep(100);
    }
  }
}

Whole example can be found at confluent github.