CODEWITHSUNDEEP
apache-kafkaapache-kafka-streamsconfluent-platform
DBSmurf
DBSmurf

Reputation: 23

Optimizing State Store topics for downstream subscription

I have a microservice using Streams 1.1.1 (planning to upgrade to 2.1 soon), that is publishing aggregate records to a compacted topic ("customer-events"), to be consumed by downstream microservices as their input KTable.

This ends up creating two topics with exact copies of the same information. “customer-events” is one; the other is the internal one created behind the scenes to back the state store. Both have the exact same keys and values.

Is there a way to either --

A) Optimize the internal topic naming convention so that we can simply use the state store backing changelog topic as the input topic for our microservices but not fear versioning will break the naming convention?

OR

B) Disable logging for the state store and if the state store has to be re-built, force it to use the "customer-events" topic as its input. (Preferred)

Upvotes: 1

Views: 175

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

The only think you could do, is to not write the data into a sink topic via to() and let the downstream consumer read directly from the changelog topic that is created anyway. If you name the KTable via Materialized.as(...) parameter, the name of the changelog topic will use it as a component of the changelog topic name (it's not possible, to specify the full name of the changelog topic). Naming the KTable provides compatibility such that the name does not change if you upgrade your application.

Reusing the output topic and omitting the changelog topic is an optimization we plan to add to Kafka Streams in the future though (cf. https://issues.apache.org/jira/browse/KAFKA-6035).

If you disable logging, you would use fault-tolerance guarantees, thus, this seems not to be an option.

Upvotes: 1

Related Questions