CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-kinesis
ShaiNe Ram
ShaiNe Ram

Reputation: 507

How to read data from s3 bucket to Kinesis Stream

I am new to Kinesis. I want to read data from s3 bucket to Kinesis stream. I have created a kinesis stream called 'mystream'. In my project every 3 month a record will be uploaded to the s3. At the same time I need to read the record to kinesis stream. I have created Lamda for reading the file and save it into the RDS DB instance. I found in lambda project there is one kinesis event. Is this the way to do the thing? please help me to get this. I don't know how to start it and all.. It would be great if some one can help me....

Upvotes: 2

Views: 17412

Answers (3)

ilminternational
ilminternational

Reputation: 19

With the AWS SDK release, you may need to change the way you're reading the S3 file into your Lambda function. The contents now come back as a stream:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_ResponseSyntax

You can stream the data into your Lambda: https://carova.io/snippets/read-data-from-aws-s3-with-nodejs

Then inside your Lambda Function, push the contents into Kinesis Pushing AWS Lambda data to Kinesis Stream

Upvotes: 0

Aref Karimi
Aref Karimi

Reputation: 1890

AWS suggests the use of AWS DMS for ingesting files from S3 to Kinsis.

https://aws.amazon.com/blogs/big-data/streaming-data-from-amazon-s3-to-amazon-kinesis-data-streams-using-aws-dms/

Upvotes: 3

Mohan Shanmugam
Mohan Shanmugam

Reputation: 690

If my understanding is right your question is about how to write s3 object data into kinesis.

You can create an event Notification in S3 for lambda. In events, you can select the event (ObjectCreate (All), put, post). After Successful creation for each object create action you get notifications in lambda. In lambda event context you get the s3 object information of each invocation.

Now you can read about the object information and write it in kinesis stream.

Links for AWS S3 and Kinesis Stream using Java

https://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html

https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html

Upvotes: 6

Related Questions