Reputation: 1893
I got this list
res22: Array[(String, List[(String, Int)])] = Array((Door_182,List((IN,1), (IN,1))), (Door_89,List((IN,1), (IN,1), (IN,1))), (Door_180,List((IN,1), (IN,1), (IN,1), (IN,1))), (Door_83,List((IN,1), (IN,1), (IN,1))), (Door_177,List((IN,1), (IN,1))), (Door_23,List((IN,1), (IN,1))), (Door_128,List((IN,1), (IN,1))), (Door_34,List((IN,1), (IN,1))), (Door_18,List((IN,1), (IN,1))), (Door_32,List((IN,1))), (Door_76,List((IN,1), (IN,1), (IN,1))), (Door_87,List((IN,1), (IN,1), (IN,1))), (Door_197,List((IN,1), (IN,1))), (Door_133,List((IN,1), (IN,1))), (Door_119,List((IN,1), (IN,1))), (Door_113,List((IN,1), (IN,1), (IN,1), (IN,1), (IN,1))), (Door_155,List((IN,1), (IN,1), (IN,1), (IN,1), (IN,1))), (Door_168,List((IN,1), (IN,1), (IN,1))), (Door_115,List((IN,1), (IN,1))), (Door_9,List((IN,1), (IN,1))),...
I tried to sum the number of IN for each door with this:
scala> reduced.map(n => (n._1, n._2)).reduceByKey((v1,v2) => v1 + v2.toString).collect
I get this error:
<console>:32: error: type mismatch;
found : List[(String, Int)]
required: String
reduced.map(n => (n._1, n._2)).reduceByKey((v1,v2) => v1 + v2).collect
^
How I can solve this?
Upvotes: 1
Views: 851
Reputation: 4515
You can do it in two steps: for each key aggregate all lists together and then sum all values in each list:
val x = sc.parallelize(List(("Door_182",List(("IN",1), ("IN",1))), ("Door_89",List(("IN",1), ("IN",1), ("IN",1))), ("Door_180",List(("IN",1), ("IN",1), ("IN",1), ("IN",1))), ("Door_83",List(("IN",1), ("IN",1), ("IN",1))), ("Door_177",List(("IN",1), ("IN",1)))))
x.reduceByKey(_ ::: _)
.map {
case (door, list) => (door, list.foldLeft(0){
case (count1, (in2, count2)) => count1 + count2
})
}.collect()
res3: Array[(String, Int)] = Array((Door_180,4), (Door_83,3), (Door_177,2), (Door_182,2), (Door_89,3))
Or in a single operation with aggregateByKey avoiding extra memory allocation:
x.aggregateByKey(0)(
{
case (count, list) => count + list.foldLeft(0){
case (count1, (in2, count2)) => count1 + count2}
},
_ +_
)
.collect()
Upvotes: 1