CODEWITHSUNDEEP
apache-sparkapache-spark-2.0sparkcore
sjuggernaut
sjuggernaut

Reputation: 1

Unable to set config in spark-submit from command line

I am trying to set master URL in the Application jar using below:

val spark = SparkSession
  .builder()
  .master("spark://master:7077")
  .appName("TestApp")
  .config("spark.sql.warehouse.dir", "/tmp/spark-warehouse")
  .getOrCreate()

I try to run the jar with below parameters:

spark-submit -—num-executors 10 -—executor-memory 5G appJar.jar

spark-submit does not take the explict options provided to the jar. When the master URL is set in application jar, every other config is ignored? Can someone explain this please?

Upvotes: 0

Views: 831

Answers (1)

koiralo
koiralo

Reputation: 23109

I think correct way is to give -- 

spark-submit --num-executors 10 --executor-memory 5G appJar.jar

Upvotes: 2

Related Questions