CODEWITHSUNDEEP
scalaapache-sparkiterator
Dr Y Wit
Dr Y Wit

Reputation: 2020

Scala function error: type mismatch

I need to calculate new column while processing data using Spark. Below code works fine

scala> def f[Int](iter: Iterator[Int]) : Iterator[(Int, Int)] = {
     |   var res = List[(Int, Int)]()
     |   while (iter.hasNext)
     |   {
     |     val x = iter.next;
     |     res = res ::: List((x, x))
     |   }
     |   res.iterator
     | }
f: [Int](iter: Iterator[Int])Iterator[(Int, Int)]

scala> sc.parallelize(Iterator(1,1,2,5,1,3).toSeq).mapPartitions(f).collect
res34: Array[(Int, Int)] = Array((1,1), (1,1), (2,2), (5,5), (1,1), (3,3))

Second value of each element should be calculated using some complex logic and defining of function fails when I try to use variable or even constant.

scala> def f[Int](iter: Iterator[Int]) : Iterator[(Int, Int)] = {
     |   var res = List[(Int, Int)]()
     |   var y = 1
     |   while (iter.hasNext)
     |   {
     |     val x = iter.next;
     |     res = res ::: List((x, y)) // List((x, 1))
     |   }
     |   res.iterator
     | }
<console>:39: error: type mismatch;
 found   : List[(Int, Any)]
 required: List[(Int, Int)]
           res = res ::: List((x, y)) // List((x, 1))

Why does it fail and how to fix? Thanks

Upvotes: 2

Views: 916

Answers (2)

Benjamin
Benjamin

Reputation: 3467

I do not know what line 39 is but why recoding map function to add a 1?

You can just do : 

val x = List(1,2,3,4,5).iterator
val y = x.map((_,1))

print(y.mkString(" "))

(1,1) (2,1) (3,1) (4,1) (5,1)

Upvotes: 0

T. Gawęda
T. Gawęda

Reputation: 16076

Delete [Int] from signature.

It tells compiler, that "Int" in the scope of the function is a type argument, so it can be any type - Int, String.

When you do var y = 1, Scala infers type scala.Int, but in your function Int if used as type argument and List is a list of tuples of this type argument, not Scala.Int.

Those two will work:

def f[E](iter: Iterator[E]) : Iterator[(E, Int)] = {
    var res = List[(E, Int)]()
    var y = 1
    while (iter.hasNext)
    {
      val x = iter.next;
      res = res ::: List((x, y)) // List((x, 1))
    }
    res.iterator
  }

and

def f(iter: Iterator[Int]) : Iterator[(Int, Int)] = {
    var res = List[(Int, Int)]()
    var y = 1
    while (iter.hasNext)
    {
      val x = iter.next;
      res = res ::: List((x, y)) // List((x, 1))
    }
    res.iterator
  }

Upvotes: 3

Related Questions