CODEWITHSUNDEEP
apache-sparkscalatest
AStack41
AStack41

Reputation: 39

How do I pass functions into Spark transformations during scalatest?

I am using Flatspec to run a test and keep hitting an error because I pass a function into map. I've encountered this problem a few times, but just found a workaround by just using an anonymous function. That doesn't seem to be possible in this case. Is there a way of passing functions into transformations in scalatest?

code:

“test” should “fail” in { 
  val expected = sc.parallelize(Array(Array(“foo”, “bar”), Array(“bar”, “qux”)))

  def validateFoos(firstWord: String): Boolean = {
    if (firstWord == “foo”) true else false
  }

 val validated = expected.map(x => validateFoos(x(0)))
  val trues = expected.map(row => true)

  assert(None === RDDComparisons.compareWithOrder(validated, trues))
}

error: org.apache.spark.SparkException: Task not serializable

*This uses Holden Karau's Spark testing base: https://github.com/holdenk/spark-testing-base

Upvotes: 0

Views: 101

Answers (1)

Holden
Holden

Reputation: 7452

The "normal" way of handing this is to define the outer class to be serilizable, this is a bad practice in anything except for tests since you don't want to ship a lot of data around.

Upvotes: 1

Related Questions