CODEWITHSUNDEEP
scalaapache-sparkspark-graphx
pauli
pauli

Reputation: 4291

scala type mismatch error in graphX code

I'm new to scala, learning it for apache-spark. I wrote a simple function in scala for graphX

def foo(edge: EdgeTriplet[Map[Long, Double], Double]): Iterator[(VertexId, Map[Long, Double])] = {

    val m = edge.srcAttr

    for((k, v) <- m){
        if (v + edge.attr < edge.dstAttr.getOrElse(k, 10.0))
            Iterator(edge.dstId, Map(k -> v + edge.attr))

        else

            Iterator.empty

    }

}

Errors

Name: Compile Error
Message: <console>:37: error: type mismatch;
 found   : Double
 required: String
                   Iterator(edge.dstId, Map(k -> v + edge.attr))
                                                          ^
<console>:35: error: type mismatch;
 found   : Unit
 required: Iterator[(org.apache.spark.graphx.VertexId, Map[Long,Double])]
    (which expands to)  Iterator[(Long, Map[Long,Double])]
           for((k, v) <- m){
                      ^
StackTrace:

Why scala is treating v as string? and what is the cause of second error?

After editing the code as suggested by @Alexey, I'm getting error 

Name: Compile Error
Message: <console>:30: error: type mismatch;
 found   : scala.collection.immutable.Map[org.apache.spark.graphx.VertexId,scala.collection.immutable.Map[Long,Double]]
    (which expands to)  scala.collection.immutable.Map[Long,scala.collection.immutable.Map[Long,Double]]
 required: Iterator[(org.apache.spark.graphx.VertexId, Map[Long,Double])]
    (which expands to)  Iterator[(Long, Map[Long,Double])]
                 (k, v) <- edge.srcAttr
                        ^
StackTrace:

If it helps, I'm implementing a different version of the sendMessage function from this code

Upvotes: 0

Views: 317

Answers (1)

Alexey Romanov
Alexey Romanov

Reputation: 170723

The first is a common problem with + in Scala, unfortunately. In this case you have not k -> (v + edge.attr), as you probably expected, but (k -> v) + edge.attr. And the only + method on a Tuple2 accepts String. To fix it, just add the correct parentheses.

The second error is because for(...) { ... } returns Unit (it's translated to a foreach call). You are missing yield. In fact if you want to use for, it should be something like

for {
  (k, v) <- m
  x <- if (v + edge.attr < edge.dstAttr.getOrElse(k, 10.0))
      Iterator((edge.dstId, Map(k -> (v + edge.attr))))
    else
      Iterator.empty[(Long, Map[Long,Double])]
} yield x

I'd prefer to write it as

m.flatMap { case (k, v) => 
  if (v + edge.attr < edge.dstAttr.getOrElse(k, 10.0))
    Iterator((edge.dstId, Map(k -> (v + edge.attr))))
  else
    Iterator.empty[(Long, Map[Long,Double])]
}

Upvotes: 1

Related Questions