CODEWITHSUNDEEP
amazon-emraws-cli
DataGuy
DataGuy

Reputation: 1725

AWS EMR --steps

I am running the following .sh to run a command on AWS using EMR:

aws emr create-cluster --name "Big Matrix Re Run 5" --ami-version 3.1.0 --auto-terminate --log-uri FILE LOCATION --enable-debugging --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=c3.xlarge InstanceGroupType=CORE,InstanceCount=3,InstanceType=c3.xlarge --steps NAME AND LOCATION OF FILE

I've deleted the pertinent file name and locations as those aren't my issue, but I am having an issue with the --steps portion of the script.

How do I specify the steps that I want to run in the cluster? The documentation doesn't give any examples.

Here is the error:

Error parsing parameter '--steps': should be: Key value pairs, where values are separated by commas, and multiple pairs are separated by spaces. --steps Name=string1,Jar=string1,ActionOnFailure=string1,MainClass=string1,Type=string1,Properties=string1,Args=string1,string2 Name=string1,Jar=string1,ActionOnFailure=string1,MainClass=string1,Type=string1,Properties=string1,Args=string1,string2

Thanks!

Upvotes: 1

Views: 1632

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269410

The documentation page for the AWS Command-Line Interface create-cluster command shows examples for using the --steps parameter.

Steps can be supplied on the command-line, or can refer to files available within HDFS or Amazon S3.

Within HDFS:

aws emr create-cluster --steps file://./multiplefiles.json --ami-version 3.3.1 --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=2,InstanceType=m3.xlarge --auto-terminate

Within Amazon S3:

aws emr create-cluster --steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,ActionOnFailure=TERMINATE_CLUSTER,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/model-build.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://mybucket/hive-ads/output/2014-04-18/11-07-32,-d,LIBS=s3://elasticmapreduce/samples/hive-ads/libs] --applications Name=Hive --ami-version 3.1.0 --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=2,InstanceType=m3.xlarge

Upvotes: 1

Related Questions