CODEWITHSUNDEEP
scalaapache-sparkscalatest
Ignacio Alorre
Ignacio Alorre

Reputation: 7605

Scala - method X overrides nothing

I am trying to perform some unit testing over a Spark Streaming application which involves DStreams.

I found it very useful the following suit: StreamingSuiteBase. It contains a method named testOperation to which you can pass an input, an operation to test and an expected output. It will verify if your expected output matches with the actual output.

The problem I am facing is during the equality verification I am getting indeed the exact same object, but wrapped into different collections:

The testOperation is defined as follows:

  def testOperation[U: ClassTag, V: ClassTag](
      input: Seq[Seq[U]],
      operation: DStream[U] => DStream[V],
      expectedOutput: Seq[Seq[V]],
      ordered: Boolean
    ) (implicit equality: Equality[V]): Unit = {
    val numBatches = input.size

    withOutputAndStreamingContext(setupStreams[U, V](input, operation)) {
      (outputStream, ssc) =>

      val output: Seq[Seq[V]] = runStreams[V](
        outputStream, ssc, numBatches, expectedOutput.size)
      verifyOutput[V](output, expectedOutput, ordered)
    }
  }

Which doesn't let me use as expected input a List(Array(myObject))

So my second option was to modify the method verifyOutput. I was planing to override it from my code, just adding a few lines to generate the List(Array(myObject)). Like this (Updated):

override def verifyOutput[V](output: Seq[Seq[V]],
                             expectedOutput: Seq[Seq[V]],
                             ordered: Boolean)
                             (implicit evidence$1: ClassTag[V], equality: Equality[V]): Unit = {
    super.verifyOutput(output, expectedOutput, ordered)
}

        //These three lines is what I am planning to add
        val sq = expectedOutput(0)
        val ssq = sq(0)
        val newOutput = Seq(Array(ssq))

        logInfo("--------------------------------")
        logInfo("output.size = " + output.size)
        logInfo("output")
        output.foreach(x => logInfo("[" + x.mkString(",") + "]"))
        logInfo("expected output.size = " + expectedOutput.size)
        logInfo("expected output")
        expectedOutput.foreach(x => logInfo("[" + x.mkString(",") + "]"))
        logInfo("--------------------------------")

        // Match the output with the expected output
        assert(output.size === expectedOutput.size, "Number of outputs do not match")
        if (ordered) {
          for (i <- output.indices)
            equalsOrdered(output(i), expectedOutput(i))

        } else {
          for (i <- output.indices)
            equalsUnordered(output(i), expectedOutput(i))
        }

        logInfo("Output verified successfully")
      }

The whole StreamingSuiteBase can be found here

But I am getting the following error in Eclipse:

method verifyOutput overrides nothing. Note: the super classes of class myClass contain the following, non final members named verifyOutput: def verifyOutput[V](output: Seq[Seq[V]],expectedOutput: Seq[Seq[V]],ordered: Boolean)(implicit evidence$1: scala.reflect.ClassTag[V],implicit equality: org.scalactic.Equality[V]): Unit

This is a simplify version of my test case:

import org.scalatest.FunSuite

class myClass extends StreamingSuiteBase with FunSuite {

  test("ExtCustProfileHbaseAPI") {

    //Here I would be generating my input and expected output

    val inputData = new myInitialObject()
    val expOutput = new myFinalObject()


    testOperation(inputData, processTest _, expOutput, ordered = false)


  }

  def processTest(input: DStream[myInitialObject]): DStream[(String,myFinalObject)] = {

        //Operation undertes
        val result = operation(input)

        result
    }

    //Here I added the override def verifyOutput[V: ClassTag]...
}

What am I doing wrong?

Upvotes: 0

Views: 2320

Answers (1)

lprakashv
lprakashv

Reputation: 1159

According to the source, the StreamingSuiteBase trait has a self type org.scalatest.Suite which means you have to extend Suite type class as well (in your case it is FunSuite) along with it otherwise it won't compile.

You can refer this: https://github.com/holdenk/spark-testing-base/wiki/StreamingSuiteBase

For more information on Scala self types, you can refer: https://docs.scala-lang.org/tour/self-types.html

You don't need a V: ClassTag the basic IDE generated overridden method I can see is:

override def verifyOutput[V](output: Seq[Seq[V]],
                             expectedOutput: Seq[Seq[V]],
                             ordered: Boolean)
                             (implicit evidence$1: ClassTag[V], equality: Equality[V]): Unit = {
    super.verifyOutput(output, expectedOutput, ordered)
}

Upvotes: 2

Related Questions