CODEWITHSUNDEEP
amazon-web-servicesamazon-kinesis
Ankur Shrivastava
Ankur Shrivastava

Reputation: 243

How to handle Kinesis Data Stream without EC2

I want to handle my Kinesis streaming data without using EC2 instance? Is there a possibility to accomplish this ie. through Lambda functions etc?

Upvotes: 1

Views: 233

Answers (1)

Matus Dubrava
Matus Dubrava

Reputation: 14512

Yes you can use Lambda service to process Kinesis streaming data. What you need to do is to create a Lambda function to process the data (the data will be available through event, first, parameter of the function).

In case of streaming data, your lambda function isn't invoked as a response to some event. Instead, Lambda service is periodically checking Kinesis for available data and then invokes your function.

For this to happen, you need to create event source mapping between your custom lambda function and Kinesis stream where you can also specify size of the batch that will be processed by lambda and its starting position.

Don't forget create proper role for your lambda function, it needs to have access to Kinesis service, so you need something like AWSLambdaKinesisExecutionRole permissions.

Another thing to consider is the batch size and how complicated your processing algorithm is. Lambda can run only for a limited time (currently 15 minutes is a maximum that you can specify), after that, it is automatically terminated by AWS. In such case, you will need to use something else than Lambda or split your lambda function into few smaller ones.

Upvotes: 2

Related Questions