CODEWITHSUNDEEP
javaapache-spark-2.0
Kiran Balakrishnan
Kiran Balakrishnan

Reputation: 257

how to create broadcast variable in spark 2(java)?

In Spark 1 we can use the below code to create a Spark broadcast variable:

SparkConf conf = new SparkConf();
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");

JavaSparkContext sc = new JavaSparkContext("local", "JavaAPISuite", conf);

ArrayList<Strin'valuesg> sampleList = new ArrayList<String>();
sampleList.add("value");

final Broadcast<ArrayList<String> broadcastVar = sc.broadcast(sampleList);

How can we do the same in Spark 2 using the API shown below?

SparkSession sc = SparkSession.setappName("SparkApp").getorcreate();
sc.sparkcontext().broadcast(T value, scala.reflect.ClassTag<T> evidence$11)

Upvotes: 1

Views: 3264

Answers (1)

kofucii
kofucii

Reputation: 7663

For example if you want to broadcast class named Test it should implement java.io.Serializable and then you do:

import scala.reflect.ClassTag;

ClassTag<Test> classTagTest = scala.reflect.ClassTag$.MODULE$.apply(Test.class);
Broadcast<Test> broadcastTest = sc.sparkcontext().broadcast(new Test(), classTagTest);

Upvotes: 4

Related Questions