CODEWITHSUNDEEP
apache-spark
K D
K D

Reputation: 103

How to get application Id/Job Id of job submitted to Spark cluster using Spark-submit command ?

I am submitting Apache Spark job using spark-submit command. I want to retrieve application Id or Job Id of the job submitted using spark-submit command. What should be the recommended way?

Upvotes: 4

Views: 8280

Answers (3)

sahil
sahil

Reputation: 21

you can get Running Streaming Job By their UUID or query name

Like this : sparkSession.streams.active.get(UUID) (where UUID is Job RunId)

Upvotes: 1

Chitral Verma
Chitral Verma

Reputation: 2855

Since it's not clear if you want it programatically in the app, i'll assume you do, You can get the yarn application id or job id (in local mode) with the following,

val sparkSession: SparkSession = ???
val appID:String = sparkSession.sparkContext.applicationId

Hope this answers your question.

Upvotes: 2

Ajay Srivastava
Ajay Srivastava

Reputation: 1181

Output of spark-submit command can be parsed to get the application id. This is the line you should be looking at -

2018-09-08 12:01:22 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180908120122-0001

appId=`./bin/spark-submit <options> 2>&1 | tee /dev/tty | grep -i "Connected to Spark Cluster" | grep -o app-.*[0-9]`
echo $appId
app-20180908120122-0001

Your use case is not clear but if you are looking for application id after job is completed then this could be helpful. This line may be different for yarn and other clusters.

Upvotes: 4

Related Questions