CODEWITHSUNDEEP
scalaapache-sparkrdd
mahboube boroumand
mahboube boroumand

Reputation: 15

How to create key-value pairs from textFile

I want to use a member in a RDD element as a key, how can i do this this is my data:

2 1
4 1
1 2
6 3
7 3
7 6
6 7
3 7

I want to create key/value pairs such that key is a element, also value is next element;

I wrote this code: 

def main(args: Array[String])
 {
   System.setProperty("hadoop.home.dir","C:\\spark-1.5.1-bin-hadoop2.6\\winutil")
   val conf = new SparkConf().setAppName("test").setMaster("local[4]")
   val sc = new SparkContext(conf)

   val lines = sc.textFile("followers.txt")
    .flatMap{x => (x.indexOfSlice(x1),x.indexOfSlice(x2))} 

}

but it is not true and it wont determine the index of elements; every two number is a line

Upvotes: 0

Views: 1496

Answers (1)

Glennie Helles Sindholt
Glennie Helles Sindholt

Reputation: 13154

Maybe I'm misunderstanding your question, but if you are simply looking to split your data into key-value pairs, you just need to do this:

val lines = sc.textFile("followers.txt").map(s => {
  val substrings = s.split(" ")
  (substrings(0), substrings(1))
})

Does this solve your problem?

Upvotes: 2

Related Questions