CODEWITHSUNDEEP
hadoopapache-spark
Dilee
Dilee

Reputation: 276

What is appName in SparkContext constructor and what is the usage of it?

I am new to Spark and I got a little confusion on this appName that is given next to master url. I have searched it through the internet but unable to pick up a clear answer. Online articles mentioned that the appName cannot be changed once it is created. So if there is only one sparkcontext is available for a single jvm what is appNames real usage....?

Can anybody clearly explain this to me.

thanks

Upvotes: 3

Views: 1346

Answers (1)

rogue-one
rogue-one

Reputation: 11587

Spark architecture has multiple components such as below.

  • Driver program: This the your bundled jar application which drives the entire application. The Driver talks to an instance of Spark Master to submit your job to the cluster

  • Spark Master: The Master acts as the single point of contact of the Cluster for the driver program. It allocates the necessary resources for the application to run which it does by negotiating with cluster manager.

  • Executors/workers: They are the workhorse in the system that actually processes your Dataframes/RDDs ie. your distributed data.

A Spark Master might be handling multiple jobs submitted by multiple driver applications concurrently. To monitor/track your specific application among hundreds of other applications running in the cluster you need an identifier. This is where appName will be helpful.

Upvotes: 4

Related Questions