CODEWITHSUNDEEP
scalaapache-sparkrdd
oikonomiyaki
oikonomiyaki

Reputation: 7951

Converting a Scala Iterable[tuple] to RDD

I have a list of tuples, (String, String, Int, Double) that I want to convert to Spark RDD.

In general, how do I convert a Scala Iterable[(a1, a2, a3, ..., an)] into a Spark RDD?

Upvotes: 11

Views: 14260

Answers (1)

GameOfThrows
GameOfThrows

Reputation: 4510

There are a few ways to do this, but the most straightforward way is just to use Spark Context:

import org.apache.spark._
import org.apache.spark.rdd._
import org.apache.spark.SparkContext._

sc.parallelize(YourIterable.toList)

I think sc.Parallelize needs a conversion to List, but it will preserve your structure, thus you will still get a RDD[String,String,Int,Double]

Upvotes: 11

Related Questions