CODEWITHSUNDEEP
pythonapache-spark
Jomonsugi
Jomonsugi

Reputation: 1279

SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243)

I see several post that contain the same error as the error that I am receiving, but none are leading me to a fix on my code. I have used this exact same code many times with no issue and now am having problems. Here is the error I receive:

y4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243).

Here is how I start my context within my python script:

spark = ps.sql.SparkSession.builder \
        .master("local[*]") \
        .appName("collab_rec") \
        .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/bgg.game_commen$
        .getOrCreate()

sc = spark.sparkContext
sc.setCheckpointDir('checkpoint/')
sqlContext = SQLContext(spark)

Please let me know if you have a suggestion.

Upvotes: 1

Views: 3662

Answers (1)

T. Gawęda
T. Gawęda

Reputation: 16076

SparkSession is the new entry point in Spark 2.x. This is a replacement for SQLContext, however it uses SQLContext in internal code.

Everything you were making with SQLContext should be possible with SparkSession.

If you really want to use SQLContext, use spark.sqlContext variable

Upvotes: 2

Related Questions