CODEWITHSUNDEEP
apache-sparkpyspark
semore_1267
semore_1267

Reputation: 1437

Possible to get output from Spark App submitted in cluster mode?

Is it possible to get output from my Spark App submitted in cluster mode? If so, how?

I'm running a simple Spark application using Python. The program just sets up a Spark Context and prints This app ran successfully to the screen. When I submit this app with the following:

spark-submit --deploy-mode client --master local[*] foo.py

it runs successfully and prints out the message.

However, when I run the same app with:

spark-submit --deploy-mode cluster --master yarn-cluster foo.py

it runs successfully, but I get no output.

While I've been using Spark for a few months now, I'm relatively new to submitting apps in cluster mode, so any help/documentation would be great!

Upvotes: 0

Views: 480

Answers (1)

Zhang Tong
Zhang Tong

Reputation: 4719

You can save This app ran successfully to external storage system such as:

sc.parallelize(['This app ran successfully'], 1).saveAsTextFile(path='hdfs:///somewhere/you/want')

Upvotes: 1

Related Questions