CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambda
Aditya Singh
Aditya Singh

Reputation: 9

How to store data into s3 bucket using an AWS SQS Queue

I have live streaming data coming my way different sources. We have created an AWS SQS Queue to get those data. I was directly pushing this incoming data to DynamoDB using lambda function but I have been asked to store the raw files into s3 bucket first. So that we do not have any data loss and later use that data for further transformations.

I have an AWS SQS Queue and I want to store the messages of this queue into an S3 bucket in JSON or Parquet form. Or is there any other alternative other than s3 to store the raw data in AWS?

I tried searching over net but couldn't find anything concrete. I am new to this please help me solve this.

Upvotes: 0

Views: 491

Answers (1)

Marcin
Marcin

Reputation: 238727

I tried searching over net but couldn't find anything concrete.

That's correct, because there is no such AWS provided feature or tool. You have to implement custom solution for that yourself.

Since you are using lambda which reads from the queue, you can add code to the lambda function to write that data to S3 before writing to DynamoDB.

Upvotes: 0

Related Questions