CODEWITHSUNDEEP
apache-kafkaksqldb
mkvakin
mkvakin

Reputation: 631

How can I get last N records from a stream using Kafka KSQL

Suppose I have a stream that contains stock prices. Each record consists of

e.g.

I need to fetch last 60 price records for AAPL. I don't want to use time windows - just last 60 records.

Upvotes: 0

Views: 1474

Answers (1)

Andrew Coates
Andrew Coates

Reputation: 1893

You can build a table that tracks the last 60 prices, per-ticker, using something like:

CREATE TABLE RECENT_PRICES AS 
  SELECT
    tickerId,
    LATEST_BY_OFFSET(price, 60) AS PRICES
  FROM TICKS
  GROUP BY tickerId;

LATEST_BY_OFFSET docs.

The PRICES column will be an ARRAY<DECIMAL(?, ?)> holding the latest 60 prices.

Note: this is latest by offset, not by timestamp. So out-of-order data may cause problems.

Upvotes: 1

Related Questions