CODEWITHSUNDEEP
amazon-web-servicesapache-sparkamazon-s3pysparkamazon-emr
ouila
ouila

Reputation: 65

Unable to spark-submit a pyspark file on s3 bucket

I have a pyspark code stored both on the master node of an AWS EMR cluster and in an s3 bucket that fetches over 140M rows from a MySQL database and stores the sum of a column back in the log files on s3.

When I spark-submit the pyspark code on the master node, the job gets completed successfully and the output is stored in the log files on the S3 bucket.

However, when I spark-submit the pyspark code on the S3 bucket using these- (using the below commands on the terminal after SSH-ing to the master node)

  1. spark-submit --master yarn --deploy-mode cluster --py-files s3://bucket_name/my_script.py This returns a Error: Missing application resource. error.

  2. spark_submit s3://bucket_name/my_script.py This shows :

20/07/02 11:26:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2857)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2896)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2878)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
    at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1911)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:766)
    at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:137)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
    at scala.Option.map(Option.scala:146)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:355)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2273)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2367)
    ... 20 more

I read about having to add a Spark Step on the AWS EMR cluster to submit a pyspark code stored on the S3.

Am I correct in saying that I would need to create a step in order to submit my pyspark job stored on the S3?

In the 'Add Step' window that pops up on the AWS Console, in the 'Application location' field, it says that I'll have to type in the location to the JAR file. What JAR file are they referring to? Does my pyspark script have to be packaged into a JAR file and how do I do that or do I mention the path to my pyspark script?

In the 'Add Step' window that pops up on the AWS Console, in the Spark-submit options, how do I know what to write for the --class parameter? Can I leave this field empty? If no, why not?

I have gone through the AWS EMR documentation. I have so many questions because I dived nose-down into the problem and only researched when an error popped up.

Upvotes: 1

Views: 6793

Answers (2)

SnigJi
SnigJi

Reputation: 1410

No its not mandatory to use STEP to submit spark job.
You can also use spark-submit

To submit a pyspark script using STEP please refer aws doc and stackoverflow

For problem 1:
By default spark will use python2. You need to add 2 config

Go to $SPARK_HOME/conf/spark-env.sh and add

export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/bin/python3

Note: If you have any custom bundle add that using --py-files

For problem 2:
A hadoop-assembly jar exists on /usr/share/aws/emr/emrfs/lib/. That contains com.amazon.ws.emr.hadoop.fs.EmrFileSystem.

You need to add this to your classpath.

A better option to me is to create a symbolic link of hadoop-assembly jar to HADOOP_HOME (/usr/lib/hadoop) in your bootstrap action.

Upvotes: 0

srikanth holur
srikanth holur

Reputation: 770

Your spark submit should be this.

spark-submit --master yarn --deploy-mode cluster s3://bucket_name/my_script.py

--py-files is used if you want to pass the python dependency modules, not the application code.

When you are adding step in EMR to run spark job, jar location is your python file path. i.e. s3://bucket_name/my_script.py

Upvotes: 1

Related Questions