CODEWITHSUNDEEP
scalaapache-sparkdatabricksazure-databricksscalatest
Sourav Chatterjee
Sourav Chatterjee

Reputation: 67

How to run a Spark-Scala unit test notebook in Databricks?

I am trying to write a unit test code for my Spark-Scala notebook using scalatest.funsuite but the notebook with test() is not getting executed in databricks. Could you please let me know how can I run it?

Here is the sample test code for the same.

import org.apache.spark.sql.{Row, SparkSession}
import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.Assertions._
import org.apache.spark.eventhubs._
import com.fasterxml.uuid.Generators
import org.apache.spark.sql.functions._
import org.apache.spark.sql.{DataFrame, Row, SparkSession, Column}

class ImsLoyaltyTest extends AnyFunSuite {
  test ("Date Format Checker") {
    val sampleIpData = Seq(
      Row("india","hello","05-03-2021","14","50"),
      Row("india","hello","15-06-2021","14","50"),
      Row("india","hello","05/06/2021","6100","50"),
      Row("india","hello","05-31-2021","14","50")
    )
    
    val sampleIpSchema = new StructType()
      .add("a", StringType)
      .add("b", StringType)
      .add("c", StringType)
      .add("d", StringType)
      .add("e", StringType)
    
    val sampleIpDF = spark.createDataFrame(spark.sparkContext.parallelize(sampleIpData), sampleIpSchema)
    
    assert (sampleIpDF.collectAsList() == sampleIpDF.collectAsList())
    
  }
}

Upvotes: 0

Views: 1568

Answers (1)

Alex Ott
Alex Ott

Reputation: 87154

You need to explicitly create the object for that test suite & execute it. In IDE you're relying on specific runner, but it doesn't work in the notebook environment.

You can use either the .execute function of create object (docs):

(new ImsLoyaltyTest).execute()

but it could be better to use the .run method of ScalaTest shell (docs) - you can control color output, run multiple tests, etc.:

import org.scalatest._

nocolor.durations.stats.run(new ImsLoyaltyTest)

Upvotes: 2

Related Questions