CODEWITHSUNDEEP
amazon-web-servicesapache-sparkamazon-emr
Alon
Alon

Reputation: 11895

How to add an EMR Spark Step?

According to the docs:

For Step type, choose Spark application.

But in Amazon EMR -> Clusters -> mycluster -> Steps -> Add step -> Step type, the only options are:

enter image description here

Upvotes: 3

Views: 7575

Answers (3)

Abdelrahman Maharek
Abdelrahman Maharek

Reputation: 862

There are two ways to add EMR spark steps:

- Using command-runner.jar (custom application) enter image description here

spark-submit --class org.apache.spark.examples.SparkPi /usr/lib/spark/examples/jars/spark-examples.jar 10

Using aws cli to do the same

aws emr add-steps --cluster-id j-xxxxxxxx --steps Name="add emr step to run spark",Jar="command-runner.jar",Args=[spark-submit,--class,org.apache.spark.examples.SparkPi,/usr/lib/spark/examples/jars/spark-examples.jar,10]

- Spark Application enter image description here

Upvotes: 3

Ajay Kr Choudhary
Ajay Kr Choudhary

Reputation: 1362

You can use command-runner.jar for your use case. For the step type let it be Custom Jar from the options that you have. Check out this image for detail.

command-runner usage

You can read more about command-runner.jar command-runner-usage

Upvotes: 1

Alon
Alon

Reputation: 11895

I don't have a Spark Application option because I created a Core Hadoop cluster.

When I created the cluster, under Software configuration, I should have chosen Spark, then I would have had the Spark application option under Step type.

Upvotes: 1

Related Questions