CODEWITHSUNDEEP
scalaimplicitscalatest
psarka
psarka

Reputation: 1842

Importing implicit methods in scalatest

I'm struggling to understand why implicit imports do not work as I expect them in scalatest. The simplified failing example (using spark, but I can make it fail with my custom class also) is as follows:

class FailingSpec extends FlatSpec with Matchers with MySparkContext {

    val testSqlctx = sqlctx
    import sqlctx.implicits._

    "sql context implicts" should "work" in {
        val failingDf = Seq(ID(1)).toDS.toDF
    }
}

The MySparkContext trait creates and destroys spark context in beforeAll and afterAll, and makes sqlctx available (already having to reassign it to a local variable in order to import implicits is a puzzle, but maybe for a different time). The .toDS and .toDF are then implicit methods imported from sqlctx.implicits. Running the test results in a java.lang.NullPointerException.

If I move import into test block things work:

class WorkingSpec extends FlatSpec with Matchers with MySparkContext {

    "sql context implicts" should "work" in {      
        val testSqlctx = sqlctx
        import sqlctx.implicits._

        val workingDf = Seq(ID(1)).toDS.toDF
    }
}

Any ideas why can't I import implicits at the top level of the test class?

Upvotes: 0

Views: 439

Answers (1)

Alfredo Gimenez
Alfredo Gimenez

Reputation: 2224

beforeAll runs before any of the tests, but does not run before the constructor for the class. The order of operations in the first snippet is:

  1. Constructor invoked, executing val testSqlctx = sqlctx and import sqlctx.implicits._

  2. beforeAll invoked

  3. Tests run

And the order of operations for the second snippet:

  1. beforeAll invoked

  2. Tests run, executing val testSqlctx = sqlctx and import sqlctx.implicits._

Assuming you give your SparkContext a default (null) value and initialize it in beforeAll, the first order of operations would try to use sqlctx when it is still null, thus causing the null pointer exception.

Upvotes: 1

Related Questions