CODEWITHSUNDEEP
amazon-web-servicesaws-serverless
pnet_fabric
pnet_fabric

Reputation: 89

which aws product to use for long-running ruby tasks execution?

in my app I'm relying on AWS a lot. I got some events coming to SNS and once they appear, I would need to fire long-running tasks. Such task is basically data extraction from multiple web services (not AWS), reconciliation of it, and persistence reconciled results in it into the database.

Because of data volume and requests throttling on the side of requested web services, I can't make it run shorter than several hours. So, I need 'some' AWS service/product which would allow me to instantiate my script in an execution-ready environment and then run it for hours. I use lambdas a lot, but they don't work in this case due to the duration of execution. Of course, I don't want anything to pay for such a service when tasks are not executed.

What products/architecture from AWS would you recommend for such a case when a serverless long-running task should be executed?

Upvotes: 0

Views: 74

Answers (1)

Mark B
Mark B

Reputation: 200988

I would create an ECS task that is deployed to Fargate. You could either have the SNS message trigger something that starts a task immediately, perhaps by building a workflow in AWS Step Functions, or just have SNS trigger a Lambda which then creates the ECS task, or you could push it from SNS to an SQS queue and have a service defined in ECS that auto-scales based on the number of messages waiting in the queue.

AWS Batch may be a good fit for this. It would manage the Fargate deployments for you and even help you utilize spot requests to save money.

Upvotes: 2

Related Questions