CODEWITHSUNDEEP
apache-spark
iamorozov
iamorozov

Reputation: 811

Spark Applications name convention

What is a name convention for a Spark Application defined by:

SparkSession
      .builder
      .appName("name goes here")

which is displayed in SparkUI?

Upvotes: 1

Views: 862

Answers (2)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29175

what ever app name you mentioned... same app name will come in the spark ui

in the above example : 

 val spark = SparkSession.builder()
      .appName("name goes here")
      .config("spark.master", "local").getOrCreate()

name displayed in the UI is ...

enter image description here

if we are talking about application id of yarn see code here, 

@Public
  @Unstable
  public static ApplicationId newInstance(long clusterTimestamp, int id) {
    ApplicationId appId = Records.newRecord(ApplicationId.class);
    appId.setClusterTimestamp(clusterTimestamp);
    appId.setId(id);
    appId.build();
    return appId;
  }

Conclusion : There is no correlation in name between appname you mentioned in the sparksession and yarn application id name which is generated by yarn.

Upvotes: 2

Ajay Ahuja
Ajay Ahuja

Reputation: 1313

You can use any name but standard name used by spark is -

e.g application_<cluster-timestamp>_<counter>

cluster-timestamp = The start-time of the "ResourceManager"
counter           = Monotonically increasing counter for the application

Upvotes: 2

Related Questions