CODEWITHSUNDEEP
scalaunit-testingapache-sparkspark-streamingscalatest
jithinpt
jithinpt

Reputation: 1244

How to write unit tests for Spark Streaming programs?

I'm relatively new to using Spark Streaming. I've been searching for the best way to write unit tests for my Spark application and came across the TestSuiteBase trait.

However, I'm unable to extend this trait in my test suite.

Here's the code snippet releant to this issue: 

... 
import org.apache.spark.rdd.RDD 
import org.apache.spark.streaming._ 
import org.apache.spark.streaming.TestSuiteBase
... 
... 
class UnitTest extends BaseTest with TestSuiteBase
...

However, I hit this error when running sbt test: 

.... object TestSuiteBase is not a member of package org.apache.spark.streaming
[error] import org.apache.spark.streaming.TestSuiteBase

Also, are there any better approaches to writing unit tests for Spark Streaming programs?

Any help would be appreciated.

Thanks in advance.

Upvotes: 1

Views: 635

Answers (1)

user2412953
user2412953

Reputation: 1

I modified my "build.sbt" to contain:

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.4.0" classifier "tests"

This will include the test jars for spark streaming, which contains TestSuiteBase

Upvotes: 0

Related Questions