CODEWITHSUNDEEP
scalaunit-testingjunitapache-spark
TheGT
TheGT

Reputation: 412

Spark (scala) Unit test - Mocking an object member

I have a spark application that involves 2 scala companion objects as follows. 

object actualWorker {
  daoClient

  def update (data, sc) {
    groupedData = sc.getRdd(data).filter. <several_operations>.groupByKey
    groupedData.foreach(x => daoClient.load(x))
  }
}


object SparkDriver {
  getArgs
  sc = getSparkContext
  actualWorker.update(data, sc : sparkContext)

}

The challenge I have is in writing unit-test for this spark application. I am using Mockito and ScalaTest, Junit for these tests.

I am not able to mock the daoClient while writing the unit test. [EDIT1: Additional challenge is the fact that my daoClient is not serializable. Because I am running it on spark, I simply put it in an object (not class) and it works on spark; but it makes it non unit-testable ]

I have tried the following:

  1. Make ActualWorker a class that can have a uploadClient passed in the Constructor. Create a client and instantiate it in Actual Worker Problem: Task not serializable exception.
  2. Introduce a trait for upload client. But still I need to instantiate a client at some point in the SparkDriver, which I fear will cause the Task Not serializable exception.

Any inputs here will be appreciated.

PS: I am fairly new to Scala and spark

Upvotes: 0

Views: 1603

Answers (1)

Jeff L
Jeff L

Reputation: 141

While technically not exactly a unit testing framework, I've used https://github.com/holdenk/spark-testing-base to test my Spark code and it works well.

Upvotes: 0

Related Questions