CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambdaamazon-data-pipeline
RAbraham
RAbraham

Reputation: 6314

Configure and Deploy Lambda Pipeline in code

I was wondering if there are any AWS Services or projects which allow us to configure a data pipeline using AWS Lambdas in code. I am looking for something like below. Assume there is a library called pipeline

from pipeline import connect, s3, lambda, deploy
p = connect(s3('input-bucket/prefix'),
            lambda(myPythonFunc, dependencies=[list_of_dependencies])
            s3('output-bucket/prefix'))
deploy(p)

There can be many variations of this idea of course. This use case assumes only one s3 bucket for e.g. There could be a list of input s3 buckets.

Can this be done by AWS Data Pipeline? The documentation I have(quickly) read says that Lambda is used to trigger a pipeline.

Upvotes: 0

Views: 166

Answers (1)

Ryan Gross
Ryan Gross

Reputation: 6515

I think the closest thing that is available is the State Machine functionality within the newly released Lambda Step Functions. With these you can coordinate multiple steps that transform your data. I don't believe that they support standard event sources, so you would have to create a standard lambda function (potentially using the Serverless Application Model) to read from S3 and trigger your State Machine.

Upvotes: 1

Related Questions