CODEWITHSUNDEEP
apache-kafkaksqldbhistorian
Mohamed Zouari
Mohamed Zouari

Reputation: 415

Use of KsqlDB as a HIstorian DB

I want to make a historian database that records and retrieves data from Kafka topics to a permanent database.

I am trying to use the KsqlDB to do it, I have trying to use tables and streams from ksql but it disappears with the death of Kafka topics (extension of the retention period). My question is "is it possible to use ksqldb as a historian for kafka? " Thanks.

Upvotes: 0

Views: 360

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

Kafka data certainly can be permanent (log.retention.ms=-1 or cleanup.policy=compact), although you don't need ksqlDB to make it so. By default, a table should be indefinitely available as it would be backed by a compacted Kafka topic

Anecdotally, the New York Times stores all their articles in a singly partitioned Kafka topic to sort their articles by time.

Note that ksqlDB is a processing engine built on Kafka Streams. It's not a database.

Upvotes: 1

Related Questions