CODEWITHSUNDEEP
scalaapache-spark
mahboube boroumand
mahboube boroumand

Reputation: 15

How can I create key-value pairs?

I have used groupByKey on my key-value pairs. Now I have this (key, Iterable). I want to make these key-value pairs from it: ((key,(one element of list)),1) for all list v.

How can I do this? How can loop on iterable list?

val lines = sc.textFile("followers.txt").map(s => {
  val substrings = s.split(" ")
  (substrings(0), substrings(1))
})
val aggrigateNeighbors = lines.groupByKey().collect().foreach(println)
val friends = aggrigateNeighbors.flatMap{x=>((k,v.hasNext()),1)}

Upvotes: 1

Views: 5446

Answers (1)

The Archetypal Paul
The Archetypal Paul

Reputation: 41769

You're nearly there.

You need to map over the iterator and make your desired entry from each

Untested, sorry, I'm not in front of the computer on which I could test it.

val lines = sc.textFile("followers.txt").map(s => {
  val substrings = s.split(" ")
  (substrings(0), substrings(1))
})
val aggregateNeighbors = lines.groupByKey()
val friends = aggregateNeighbors.flatMap{case (k,v) => v.map{s=>((k,s), 1)}}

Upvotes: 2

Related Questions