CODEWITHSUNDEEP
amazon-kinesisamazon-kinesis-firehose
user2456976
user2456976

Reputation: 1308

Kinesis Consumption Lag Monitoring

I am trying to collect metrics for Kinesis at the stream level, more specifically want to get the count by which a worker/consumer process is lagging behind at the stream level, I don't care about the lags at the shard level. I found an example on calculating consumption lag on AWS docs, but it calculates the lag time, not the lag by message count. I am trying to use the Java KCL to gather these metrics for a stream, these metrics will be collected periodically and published to a dashboard. Thanks.

Upvotes: 4

Views: 1609

Answers (1)

Daniel Vassallo
Daniel Vassallo

Reputation: 344581

The lag size by number of records is not a metric that you get out of the box. You could calculate it yourself, but it's not straightforward.

The producer knows nothing about the consumer and the consumer knowns nothing about the producer. The only reason why the lag time is available is that the sequence number of each record has the ingested timestamp encoded in it, so the consumers can publish the latest timestamp they saw, and then you can get the max lag for the entire stream.

Upvotes: 1

Related Questions