CODEWITHSUNDEEP
pythonapache-sparkpyspark
Markus
Markus

Reputation: 3782

How to access SparkContext from SparkSession instance?

I am importing SparkSession as follows in PySpark:

from pyspark.sql import SparkSession

Then I create SparkSession:

spark = SparkSession.builder.appName("test").getOrCreate()

and try to access SparkContext:

spark.SparkContext.broadcast(...)

However, I get an error that SparkContext does not exist. How can I access it in order to set broadcast variables?

Upvotes: 30

Views: 38994

Answers (2)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39830

Asumming you have a spark session

spark_session = SparkSession \
    .builder \
    .enableHiveSupport() \
    .getOrCreate()

Spark Context can be inferred using

spark_context = spark_session._sc

or

spark_context = spark_session.sparkContext

Upvotes: 5

Roberto Congiu
Roberto Congiu

Reputation: 5213

You almost got it right, it's lowercase s at the beginning:

>>> spark.sparkContext
<SparkContext master=local[*] appName=PySparkShell>

Upvotes: 49

Related Questions