CODEWITHSUNDEEP
pyspark
Haripriyan V
Haripriyan V

Reputation: 1

Name sc is not defined

I am just trying to execute sc.version inside pyspark shell however getting an error as sc not defined.

>>> sc.version()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

If i run SparkContext.getOrCreate()

>>> SparkContext.getOrCreate()
<pyspark.context.SparkContext object at 0x7f206aa8cfd0>

I am not getting even the output of sc.version(). What is the problem?

Upvotes: 0

Views: 879

Answers (1)

PiyushC
PiyushC

Reputation: 308

A few things:

  1. Inside the pyspark shell you automatically only have access to the spark session (which can be referenced by "spark").
  2. To get the sparkcontext, you can get it from the spark session by sc = spark.sparkContext. Or using the getOrCreate() method as mentioned by @Smurphy0000 in the comments
  3. Version is an attribute of the spark context. To get the version from sparkcontext (sc in this case), version = sc.version. Version can also be extracted from the session directly as version = spark.version

Upvotes: 1

Related Questions