CODEWITHSUNDEEP
amazon-web-servicesarchitectureaws-lambdaaws-batch
Eliot
Eliot

Reputation: 137

Use AWS step functions for processing large amount of data?

We want to use AWS step function to processing a large amount of data from a CSV file but we are not sure if this is the best choice.

Our use case is below : - We upload a CSV with a large amount of lines (Like 50K) and for each line we process a small traitements (Each traitement is processed by a lambda function). At this time, we think the best choice is to insert each line from our CSV in a DynamoDB and for each line launch our lambda functions.

What do you think of this ?

Upvotes: 1

Views: 1787

Answers (1)

Ashan
Ashan

Reputation: 19728

There are multiple patterns to process large files with Lambda.

  • One approach is to use a Lambda function is to split the large file and delegate the parts to worker Lambda functions.
  • If the processing steps for parts are complex enough you can trigger multiple Step function workflows.

In your proposed approach, if each item processing is large enough it will make sense to process item by item, but generally its more efficient to process as batches.

Upvotes: 2

Related Questions