CODEWITHSUNDEEP
scala
theMadKing
theMadKing

Reputation: 2074

Scala Counter in a Loop causing issues at run time

I have the following function that worked fine in the REPL, essentially what it is doing is checking the datatype of a schema and matching it to the column when I flatten the file out later and zipWithIndex:

 //Match a Schema to a Column value
    def schemaMatch(x: Array[String]) = {
      var accum = 0
      for(i <- 0 until x.length) {
        val convert = x(i).toString.toUpperCase
        println(convert)
        val split = convert.split(' ')
        println(split.mkString(" "))
        matchTest(split(1), accum)
        accum += 1
      }
      def matchTest(y:String, z:Int) = y match{

        case "STRING" => strBuf += z
        case "INTEGER" => decimalBuf += z
        case "DECIMAL" => decimalBuf += z
        case "DATE" => dateBuf += z
      }
    }


    schemaMatch(schema1)

The error I am getting:

Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.IntRef.create(I)Lscala/runtime/IntRef;
    at com.capitalone.DTS.dataProfiling$.schemaMatch$1(dataProfiling.scala:112)
    at com.capitalone.DTS.dataProfiling$.main(dataProfiling.scala:131)
    at com.capitalone.DTS.dataProfiling.main(dataProfiling.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Line 112:

var accum = 0

Any ideas why its no longer working when compiled but worked in the REPL, and how to correct it?

Upvotes: 0

Views: 131

Answers (1)

Peter Neyens
Peter Neyens

Reputation: 9820

It is difficult to see the underlying problem causing the NoSuchMethodError, with the code you have provided us, but your method to extract the column types and the corresponding indices can be simplified:

def schemaMatch(schema: Array[String]) : Map[String,List[Int]] =
  schema
    // get the 2nd word (column type) in upper cases
    .map(columnDescr => columnDescr.split(' ')(1).toUpperCase)
    // combine column type with index
    .zipWithIndex
    // group by column type
    .groupBy{ case (colType, index) => colType }
    // keep only the indices
    .mapValues( columsIndices => columsIndices.map(_._2).toList )

Which can be used as:

val columns = Array("x string", "1 integer", "2 decimal", "2015 date") 
val columnTypeMap = schemaMatch(columns)
//Map(DATE -> List(3), STRING -> List(0), DECIMAL -> List(2), INTEGER -> List(1))

val strIndices = columnTypeMap.getOrElse("STRING", Nil)
// List(0)

val decimalIndices = columnTypeMap.getOrElse("INTEGER", Nil) :::
  columnTypeMap.getOrElse("DECIMAL", Nil)
// List(1, 2)

Upvotes: 1

Related Questions