CODEWITHSUNDEEP
rubyamazon-web-servicesamazon-data-pipeline
RidingRails
RidingRails

Reputation: 792

AWS Datapipeline to Ruby Code

I am in the process of taking over a set of data pipelines on AWS. They are all built using the AWS graphical editor tool. The pipelines are getting complex and my goal is move them to code and have them versioned. We are a ruby shop so besides of the AWS ruby gem is there another tool or framework I should use to do this? Any resources, blogs, docs I can follow that would be helpful and easy to follow (aws docs could be better).

Upvotes: 0

Views: 84

Answers (1)

Brian R Armstrong
Brian R Armstrong

Reputation: 410

Datapipeline supports a JSON definition format. The format is described in these links:

From the Architect, you can export your existing pipelines as JSON by following the instructions here.

Commit these JSON files to your source control, and run/migrate/deploy them using the AWS CLI like this:

aws datapipeline create-pipeline --name mydpl --unique-id mydpl
aws datapipeline put-pipeline-definition --pipeline-id df-xxxxxxxxxx --pipeline-definition file:///mydpl.json
aws datapipeline activate-pipeline --pipeline-id df-xxxxxxxxxx

Upvotes: 1

Related Questions