CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqsamazon-kinesis
JHH
JHH

Reputation: 9315

Async background jobs with AWS lambda

I want to create a REST API that lets a caller submit input for a job which will run in the background for potentially a minute or so. Hence, the API should spawn a background job and immediately return a job ID. The client may then query another endpoint for the job status and result.

The first thought that pops up in my head is something like this:

For this to work, the worker lambda has to have a timeout sufficient to finish a large job. The API lambdas will always be quick since they just make a database operation and a submission to SQS.

Still it kind of bugs me that I need to have 2 lambdas. But if I understood it correctly, it's not possible to continue executing a lambda function after it has returned a response to the caller. I also considered perhaps using Kinesis rather than SQS for the events, as order of execution is not critical, every caller just wants his job finished in a reasonable amount of time. My trafffic pattern could be bursts of 100.000 jobs and then nothing for days. I'm also considering supporting multiple sub-jobs in each job, so that each job would actually be say 1-100 units of work.

I don't want to host a instances for the workers, e.g. containers on ECS; I want to rely only on serverless concepts.

Is this a decent setup or is there a better one?

Upvotes: 5

Views: 6708

Answers (2)

Traycho Ivanov
Traycho Ivanov

Reputation: 3217

Your approach with SQS sounds good - 100.000 is not big drama and I would go fully serverless even using dynamodb as data storage.

Sure 2 lambdas, one for processing the HTTP request and second for doing the work. Make sure you enable provisioned concurrency to handle properly your expected traffic - https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/

About Kinesis ,it does not sound the perfect case for your scenario as you could have multiple days without any traffic but kinesis stream is billed per hour, you could not switch it off but delete.

I would stick to the plan to be as simple as possible and try something else if any blocker pops up.

Upvotes: 3

maafk
maafk

Reputation: 6896

This looks like a good use case for AWS Step Functions.

You can start a State Machine execution from API Gateway which will return the execution ID.

That execution ID can be used to query the status of the execution, which is doing all the necessary background processing

Upvotes: 3

Related Questions