query/filter the events in kinesis stream

Question

I am working on Kinesis-streams for my event-sourcing architecture, after failing to use kafka because of its ridiculous infrastructure need.

Anyway, I have Kinesis producers, consumers working fine(inspite of high latency). I have been doing my component-testing consuming the stream to verify that an event is written to the stream.

But what I'm looking for is a Stream dashboard, which can be used by non-technical people.

Because in production, people like managers/ (even lazy debuggers), obviously do not want to write code to view the events in stream, instead would be helpful to see in a dashboard.

I don't see the Kinesis-streams have a querying capability.

As a solution for this capability,

• I can consume an event as soon as it is written to kinesis and apply to another data-store that supports querying/filtering capabilities.(eg, MongoDB, DynamoDB), which takes time for me to implement in a pressured timeline.

• Or, streaming kinesis events to kinesis-analytics can solve my problem too, though Analytics is not designed for that purpose. And its super slow, takes upto 15/20 secs to appear in dashboard, neither does it show new events.

I tried importing the source kinesis-stream ComponentTest_Producer_Stream_2_6_117 which have JSON events in format {"eventType": "KinesisEvent", "someField" : "some value", "createdTime" : "some data"}, to kinesis-analytics.

Somehow it auto detected the events and populates in dashboard, (after multiple tries)

• I'm also not sure if Cloudwatch can solve this problem and my question completely does not make sense? Too many tools around this Amazon WS world, I'm super new to it.

My questions are

1) What data-store Kinesis-stream uses for storing the events?

2) What strategies are recommended for querying/ filtering events in a kinesis-stream.

Answer 1

If you are using lambda with event source mapping now you can add filter criteria.

Take a look:

https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/

And

https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html

Answer 2

I think the main problem that you are trying to solve is how to build a dashboard that your teammates can use to quickly peek into the system. I can suggest two options:

• Run a separate consumer application that processes data and pushes to a data store such as DynamoDB. You can build a lightweight UI that surfaces this information. One sort-of-relevant example can be found here.
• Load Kinesis data into an Elasticsearch cluster (relevant documentation here) and use Kibana for visualization. Kibana is open-source, supremely powerful and allows easy custom aggregations on the UI.