CODEWITHSUNDEEP
scalagroup-byrdd
Choix
Choix

Reputation: 575

How to transform a CompactBuffer to List

How can I transform a CompactBuffer to List? an example is as below:

data.csv
1,Lokesh
2,Bhupesh
2,Amit
2,Ratan
2,Dinesh
1,Pavan
1,Tejas
2,Sheela
1,Kumar
1,Venkat
val r = sc.textFile("data96/data.csv")
val rm = r.map(x=>(x.split(",")(0), x.split(",")(1)))

val r_grp = rm.groupByKey

Array[(String, Iterable[String])] = Array((2,CompactBuffer(Bhupesh, Amit, Ratan, Dinesh, Sheela)), (1,CompactBuffer(Lokesh, Pavan, Tejas, Kumar, Venkat)))

r_grp.foreach(println(_))
(1,CompactBuffer(Lokesh, Pavan, Tejas, Kumar, Venkat))
(2,CompactBuffer(Bhupesh, Amit, Ratan, Dinesh, Sheela))

So, it is easy to get the CompactBuffer format. The following will generate the result in List format, which is what I want to present.

val swapped = rm.map(item=>item.swap)

val com= rm.combineByKey(List(_), (x:List[String], y:String) =>y::x,(x:List[String], y:List[String])=>x:::y) //this is terrible to me, I don't want to do this way

Array[(String, List[String])] = Array((2,List(Dinesh, Ratan, Amit, Bhupesh, Sheela)), (1,List(Lokesh, Venkat, Kumar, Tejas, Pavan)))

com.repartition(1).saveAsTextFile("data96/s43")

Generated result: 

hdfs dfs -cat data96/s43/*
(2,List(Dinesh, Ratan, Amit, Bhupesh, Sheela))
(1,List(Lokesh, Venkat, Kumar, Tejas, Pavan))

It's much easier to get the result in CompactBuffer format, but I need to get List format and don't want to go through the highlighted terrible confusing command.

By the way, how do I further sort each group in asc or desc?

Upvotes: 0

Views: 3507

Answers (1)

koiralo
koiralo

Reputation: 23109

You can simply change CopmpactBuffer to list with toList method 

val r_grp = rm.groupByKey
// After groupBykey
(2,CompactBuffer(Bhupesh, Amit, Ratan, Dinesh, Sheela))
(1,CompactBuffer(Lokesh, Pavan, Tejas, Kumar, Venkat))

Simply use toList to convert to List and sortBy to sort it 

val com = r_grp.map(x => (x._1, x._2.toList.sortBy(x => x)))
com.foreach(println)

Output :

(1,List(Kumar, Lokesh, Pavan, Tejas, Venkat))
(2,List(Amit, Bhupesh, Dinesh, Ratan, Sheela))

Upvotes: 3

Related Questions