CODEWITHSUNDEEP
amazon-web-servicesapache-sparkamazon-s3pysparkamazon-emr
Gopala
Gopala

Reputation: 10473

amazon emr spark submission from S3 not working

I have a cluster up and running. I am trying to add a step to run my code. The code itself works fine on a single instance. Only thing is I can't get it to work off S3.

aws emr add-steps --cluster-id j-XXXXX --steps Type=spark,Name=SomeSparkApp,Args=[--deploy-mode,cluster,--executor-memory,0.5g,s3://<mybucketname>/mypythonfile.py]

This is exactly what examples show I should do. What am I doing wrong?

Error I get:

Exception in thread "main" java.lang.IllegalArgumentException: Unknown/unsupported param List(--executor-memory, 0.5g, --executor-cores, 2, --primary-py-file, s3://<mybucketname>/mypythonfile.py, --class, org.apache.spark.deploy.PythonRunner)

Usage: org.apache.spark.deploy.yarn.Client [options]
Options:
  --jar JAR_PATH           Path to your application's JAR file (required in yarn-cluster
                           mode)
    .
    .
    .
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Command exiting with ret '1'

When I specify as this instead:

aws emr add-steps --cluster-id j-XXXXX --steps Type=spark,Name= SomeSparkApp,Args=[--executor-memory,0.5g,s3://<mybucketname>/mypythonfile.py]

I get this error instead:

Error: Only local python files are supported: Parsed arguments:
  master                  yarn-client
  deployMode              client
  executorMemory          0.5g
  executorCores           2

EDIT: IT gets further along when I manually create the python file after SSH'ing into the cluster, and specifying as follows:

aws emr add-steps --cluster-id 'j-XXXXX' --steps Type=spark,Name= SomeSparkApp,Args=[--executor-memory,1g,/home/hadoop/mypythonfile.py]

But, not doing the job.

Any help appreciated. This is really frustrating as a well documented method on AWS's own blog here https://blogs.aws.amazon.com/bigdata/post/Tx578UTQUV7LRP/Submitting-User-Applications-with-spark-submit does not work.

Upvotes: 3

Views: 4804

Answers (2)

Adam Caron
Adam Caron

Reputation: 401

I searched for days and finally discovered this thread which states

PySpark currently only supports local files. This does not mean it only runs in local mode, however; you can still run PySpark on any cluster manager (though only in client mode). All this means is that your python files must be on your local file system. Until this is supported, the straightforward workaround then is to just copy the files to your local machine.

Upvotes: 0

V. Samma
V. Samma

Reputation: 2608

I will ask, just in case, you used your correct buckets and cluster ID-s?

But anyways, I had similar problems, like I could not use --deploy-mode,cluster when reading from S3. When I used --deploy-mode,client,--master,local[4] in the arguments, then I think it worked. But I think I still needed something different, can't remember exactly, but I resorted to a solution like this:

Firstly, I use a bootstrap action where a shell script runs the command:

aws s3 cp s3://<mybucket>/wordcount.py wordcount.py

and then I add a step to the cluster creation through the SDK in my Go application, but I can recollect this info and give you the CLI command like this:

aws emr add-steps --cluster-id j-XXXXX --steps Type=CUSTOM_JAR,Name="Spark Program",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=["spark-submit",--master,local[4],/home/hadoop/wordcount.py,s3://<mybucket>/<inputfile.txt> s3://<mybucket>/<outputFolder>/]

Upvotes: 2

Related Questions