CODEWITHSUNDEEP
scalaapache-spark
Emil
Emil

Reputation: 23

Read and transform data from textFile in Spark

I am using scala, I have a case class Point(x: Double = 0, y: Double = 0) and I am reading a textFile val data = sc.textFile("Dataset.txt").flatMap(line => line.split(",")) and the data is in the following format(each diferrent value is separated by newline in the file):

133.559,244.112

88.242,100.222

I want the key to be of type Point but after the split the numbers get separated and I don't know how to merge every two of them together to form a Point Key in the RDD. Maybe there is a way to transform it without splitting?

Thanks

Upvotes: 1

Views: 434

Answers (1)

baitmbarek
baitmbarek

Reputation: 2518

You should not flatMap :

val rdd: RDD[String] = spark.sparkContext.parallelize(Seq(
    "133.559,244.112","88.242,100.222"
  ))

  val result: RDD[Point] = rdd.map{ e =>
    val Array(x,y) = e.split(",").map(_.toDouble)
    Point(x,y)
  }

Upvotes: 1

Related Questions