CODEWITHSUNDEEP
javapythonjarpysparkspark-submit
user2549980
user2549980

Reputation: 691

Submitting Python file and Java jars to Spark- Error: No main class set in JAR; please specify one with --class

I have to execute a python file with user defined Java functions using py4j. I am using a spark-submit command to send the pyfiles and jars to the executors. The code I've written for Py4j will handle the rest.

However I received an error

Error: No main class set in JAR; please specify one with --class

The command is as follows:

 spark-submit  --files /home/USER/PATH/SOMEFILE.txt --conf spark.executor.extraClassPath= /home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar  --executor-memory 512m  --driver-class-path /home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar  --master local[*]  --jars /home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar  --driver-memory 512m  --py-files /home/USER/PATH/eggs/kafka.egg,/home/USER/PATH/eggs/redis.egg,/home/USER/PATH/helloworld.egg,/home/USER/PATH/helloworld.py

My problem is that I don't have a Java class to specify. I have python files, so what am I supposed to exactly specify?

Also I have used the --jars, --conf spark.executor.extraClassPath, --driver-class-path options to send the jar to the executors as it contains the user defined functions.

Simply mentioning --jars or --driver-class-path wasn't doing the job so tried the third parameter, which raised the error.

Thanks in advance.

Upvotes: 1

Views: 2265

Answers (1)

user2549980
user2549980

Reputation: 691

There was a space between "--conf spark.executor.extraClassPath=" and "/home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar" as seen below-

--conf spark.executor.extraClassPath= /home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar

Also moved the --conf settings to the start of the spark-submit command, like so:

spark-submit --conf spark.executor.extraClassPath=/home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar   --files /home/USER/PATH/SOMEFILE.txt --executor-memory 512m  --driver-class-path /home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar  --master local[*]  --jars /home/USER/DEFINED_FUNCTIONS/IN JAVA/XYZ/XYZ.jar  --driver-memory 512m  --py-files /home/USER/PATH/eggs/kafka.egg,/home/USER/PATH/eggs/redis.egg,/home/USER/PATH/helloworld.egg,/home/USER/PATH/helloworld.py

This sorted my issue.

Upvotes: 1

Related Questions