CODEWITHSUNDEEP
elastic-map-reduceamazon-emramazon-data-pipeline
Zach Moshe
Zach Moshe

Reputation: 2980

How to set instance role for EMR clusters launched via data pipeline?

I'm trying to attach an instance role to a cluster I'm running through data-pipeline. I'd like to run my own mapper script that needs write permissions to DynamoDB (the "regular" HIVE upload won't do the trick for me).

I've gone through the API docs and it seems that while EMR can specify an instance role for the machines, in the data-pipeline API it's not supported.

Asking here to confirm this, and would love to hear if you have other solutions..

Upvotes: 3

Views: 211

Answers (1)

panther
panther

Reputation: 767

Update:

Data Pipeline now supports creation of clusters via role. Take a look at this documentation.

Data Pipeline does not support creation of EMR clusters using role but you can still use EMR cluster. Your credentials are available based on the cluster based on your IAM user policy. So just add write permissions to DynamoDB to your IAM user policy, start pipeline using that user and then use EMR cluster in it.

Upvotes: 1

Related Questions