AWS SQS triggering from lambda to Amazon Lightsails

Question

I am a newbie to AWS world and i have been stuck in a situation where i want to send a download file URL to amazon lightsails server which is a light version of ec2 and from there the file will get downloaded to s3 .

Now here there might be a lot of download file URL generated and transmitted by lambda but my amazon lightsails server cannot handle all those large operations simultaneously and for that i thought of using AWS SQS

Where i will transfer data(download URL, credentials for file upload to s3) to my aws SQS and from their SQS will make a queue of maximum 10 entries and then will trigger and send data in a synchronous manner to amazon lightsails server so that there wont be any choked up or bottle neck condition occurring on server end . Any idea how can that be achieved ?

Thanks in Advance !

Answer 1

You described a common scenario where AWS Lambda is not suitable as a final cumputing service due to the unpredictable ammount of time & disk consumed. Lightsails (EC2) must be used instead.

You may use Lambda in order to send new messajes/jobs to SQS using any AWS SDK. Your Lightsails server now must to poll for pending messages in your SQS queue.

If you have some limitations related to the length of your queue or the retention period or any scale related problem, you can again use a scheduled lambda function in order to inspect your queue and scale horizontally as many Lightsails (EC2) instances it requires.

Here you can find a useful post. Your solution may looks like this:

Answer 2

AWS SQS cannot send(push) data to servers. From your lightsails server you will have to poll the SQS.

So in your lightsails server you can poll for a batch of say 10 messages (based on your processing capability) and then when you are through processing that poll for the next batch of requests