CODEWITHSUNDEEP
apache-spark
Jack
Jack

Reputation: 5870

Spark Broadcast Variables can not put into a function, otherwise threw NullPointerException

I found when I put broadcast variable into a function, then the function can not get the values instead of a null pointer, the code as following:

def main(){
  val serviceSet=Set("abc","ccc")
  broadServiceSet = sparkContext.broadcast(serviceSet)
  var keyRDD=esRdd.map(tp=>(map2Key(broadServiceSet.value),tp._1))
}

def map2Key(broadSet:Set[String]):String={
  broadSet.contains("testName")//here threw NullPointerException
}

Upvotes: 0

Views: 496

Answers (1)

user5314006
user5314006

Reputation:

From your code everything is good, it should not throw NullPointerException. The broadcast variables can be passed to function or other classes very well. I guess you may defined your broadcast variable as a Object field. If so, it would throw Exception, because the JVM compiler will override you did in main function instead of using the initial value which was previously defined as a class field.

Upvotes: 1

Related Questions