Reputation: 721
Question 3: I want to convert my scala map to RDD, how would I do in the following case? I am trying to do in this way
var mapRDD = sc.parallelize(map.toList)
Is this the right way to do ?
I have not included the whole code, basically mapAgainstValue contains userId as key and list of friends as values. I want to recreate a map RDD with the following transformation in the key. What would be the reason for empty map?
var mapAgainstValue = logData.map(x=>x.split("\t")).filter(x => x.length == 2).map(x => (x(0),x(1).split(",")))
var map:Map[String,List[String]] = Map()
var changedMap = mapAgainstValue.map{
line =>
var key ="";
for(userIds <- line._2){
if(line._1.toInt < userIds.toInt){
key =line._1.concat("-"+userIds);
}
else {
key = userIds.concat("-" + line._1);
}
map += (key -> line._2.toList)
}
}
changedMap.collect()
map.foreach(println)
Upvotes: 3
Views: 4902
Reputation: 8967
Yes, you can use Tuple as a key in Map.
For example:
val userMap = Map(
(1, 25) -> "shankar",
(2, 35) -> "ramesh")
Then you can try print the output using foreach
val userMapRDD = sparkContext.parallelize(userMap.toSeq, 2)
mapRDD.foreach(element => {
println(element)
})
If you want to transform the mapRDD to something else. following code returns only age and name as tuple.
val mappedRDD = userMapRDD.map {
case ((empId: Int, age: Int), name: String) => {
(age, name)
}
}
Upvotes: 5