CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Chirag Visavadiya
Chirag Visavadiya

Reputation: 567

Difference between sqlContext.sql and sqlContext.setConf

What is difference between sqlContext.sql and sqlContext.setConf, and which one should I prefer?

scala> sqlContext.sql("set spark.sql.shuffle.partitions=15")

scala> sqlContext.setConf("spark.sql.shuffle.partitions", "10")

Upvotes: 0

Views: 2573

Answers (1)

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41957

sqlContext.setConf("spark.sql.shuffle.partitions", "10") will set the property parameter for whole application before logicalPlan is generated.

sqlContext.sql("set spark.sql.shuffle.partitions=15") will also set the property but only for particular query and is generated at the time of logicalPlan creation.

Choosing between them depends on what your requirement is.

Upvotes: 3

Related Questions