CODEWITHSUNDEEP
apache-spark
Rajesh Thakkar
Rajesh Thakkar

Reputation: 73

Iterating over CompactBuffer in Spark

I have a PairedRDD where values are grouped based on key, from which I fetch a particular key:

pairsgrouped.lookup("token")

Output:

res6: Seq[Iterable[String]] = ArrayBuffer(CompactBuffer(EC-17A5206955089011B, EC-17A5206955089011A))

I want to iterate over these values and filter results even further. But I am not able to iterate over it.

I tried the following way:

pairsgrouped.lookup("token").foreach(println)

But this gives me this:

CompactBuffer(EC-17A5206955089011B, EC-17A5206955089011A)

I want to iterate over these values and filter the results using these values.

Upvotes: 1

Views: 6007

Answers (1)

Glennie Helles Sindholt
Glennie Helles Sindholt

Reputation: 13154

The values of your grouped data is an Iterable, not a list of values. You must collect the values first before you can print them. To get only the values for the group with key="token", you do an initial filtering. 

val result = groupedData.filter{case(key, _) => key == "token"}
                        .values
                        .flatMap(i => i.toList)
                        .collect()

result foreach println

Upvotes: 4

Related Questions