spark-submit job is not working on cluster mode

Question

I'm trying to execute a python file using spark-submit on cluster mode. Can someone suggest me to What I need to do before doing that as its giving error?

I have tried master as yarn and without cluster mode its running fine.

test_cluster.py:

from pyspark.sql import SparkSession
spark= 
SparkSession.builder().appName("test").master("yarn").getOrCreate()

execute command:

 spark-submit --master yarn --deploy-mode cluster test_cluster.py

YARN log:

Application application_1557254378595_0020 failed 2 times due to AM Container for appattempt_1557254378595_0020_000002 exited with exitCode: 13 Failing this attempt.Diagnostics: [2019-05-07 22:20:22.422]Exception from container-launch. Container id: container_1557254378595_0020_02_000001 Exit code: 13 [2019-05-07 22:20:22.424]Container exited with a non-zero exit code 13. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 19/05/07 22:20:21 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User application exited with status 1 19/05/07 22:20:21 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498) at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: org.apache.spark.SparkUserAppException: User application exited with 1 at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:106) at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)

Answer 1

Unfortunately, cluster deployment mode is not available for Python applications, as documented here.

You can try client deployment-mode.