CODEWITHSUNDEEP
apache-kafkadistributed-computingscalabilityapache-kafka-streamskafka-interactive-queries
best wishes
best wishes

Reputation: 6634

What,Where is the Use of Kafka Interactive Queries

Kafka can be used as an event store. and in my understanding Interactive Queries were brought into picture to solve the issue of aggregating the result when some events are still inflight (hidden in kafka topic, yet to be processed).

Let's say we want to calculate number of views on a page. Whenever a user visits we will create an event and insert in the kafka topic. And we will define Kafka Interactive Queries to aggregate the result.

But isn't it against the philosophy that we should precompute whatever we need instead of doing runtime aggregation? How does it solves race condition where new page view came while we were aggregating the results.

It is still okay in the case of page view calculation (where number of view MAY NOT have very significant impact) but let's say we start using it for payment system as claimed by many. Here are the problems that I see

  1. What to do with the existing balance, we can not create events out of them now, right?

  2. Let's say we decide to aggregate result runtime by only using Interactive Queries, how do we determine balance at any given point of time? (IMO we are still relying on eventual consistency.)

  3. How would it scale if I have to calculate aggregated balance over last 15 years? instead of storing precomputed value.

  4. If I keep the aggregated result in separate store (Let's say B), and whenever I have to compute balance I will make use of both B and interactive queries to know the current balance. But it is still prone to race conditions right?

  5. How would it be different if I just create a new record every-time in a nosql database and do runtime aggregation? Only that we wont be able to ensure transaction between credit and debit?

  6. What would happen if event schema changes?

Beside the above questions,

The only good part that I see is, We can aggregate the data differently as and when need arises.

Basically the question is what is the right use case of Interactive Queries? What are the pitfalls to avoid.

Upvotes: 1

Views: 270

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

And we will define Kafka Interactive Queries to aggregate the result.

That is not correct. You would use IQ to lookup the result (not to compute it). The computation is done by a Kafka Streams application as an aggregation, and IQ is used to interactively "tap" into the KTable state store to get the current result.

Upvotes: 2

Related Questions