CODEWITHSUNDEEP
unit-testinghadoopapache-spark
Konstantin Kudryavtsev
Konstantin Kudryavtsev

Reputation: 592

Run spark unit test on Windows

I'm trying to run some transformation on Spark, it works fine on cluster (YARN, linux machines). However, when I'm trying to run it on local machine (Windows 7) under unit test, I got errors:

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)

My code is following:

@Test
def testETL() = {
    val conf = new SparkConf()
    val sc = new SparkContext("local", "test", conf)
    try {
        val etl = new IxtoolsDailyAgg() // empty constructor

        val data = sc.parallelize(List("in1", "in2", "in3"))

        etl.etl(data) // rdd transformation, no access to SparkContext or Hadoop
        Assert.assertTrue(true)
    } finally {
        if(sc != null)
            sc.stop()
    }
}

Why is it trying to access hadoop at all? and how can I fix it? Thank you in advance

Upvotes: 0

Views: 2426

Answers (1)

Konstantin Kudryavtsev
Konstantin Kudryavtsev

Reputation: 592

I've solved this issue on my own http://simpletoad.blogspot.com/2014/07/runing-spark-unit-test-on-windows-7.html

Upvotes: 2

Related Questions