CODEWITHSUNDEEP
javamapreduceapache-spark
YuliaSh.
YuliaSh.

Reputation: 795

Spark: get JavaSparkContext that was created in another class

I have spark application consisting of several classes. I create JavaSparkContext in my main class and use it for getting data from hdfs (sc.textFile). Then this data is passed to another object that is processing it. During the processing I need the context again to make RDD from List. I don't want to add one more parameter in the API to pass context object from main class to the another one. Are there any other options, like getting the existing context using some static method?

P.S. Searching in the web didn't help.

Upvotes: 2

Views: 1636

Answers (1)

maasg
maasg

Reputation: 37435

You create a SparkContext instance by providing a configuration to the class constructor. Therefore, if you want to use that instance in another part of your program, you need to pass that same reference in some way.

Passing the context around as a parameter to the jobs is a valid option and would be my preferred way. Make sure you make that parameter transient to avoid it being 'caught' by a closure.

You could also use a classical 'Singleton' pattern using a class that encapsulates the context creation and provides a static method to get the singleton reference.

In all cases, you need to provide the ctx sharing mechanism that fits your specific needs.

Upvotes: 3

Related Questions