CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
AMARNATH SHETTY V
AMARNATH SHETTY V

Reputation: 13

Scala And Spark , rdd to dataframe creation from of dictionary

Can you please let me know on how to create data frame from the following code ?

val x =List(Map("col1"->"foo","col2"->"bar"))
val RDD =sc.parallelize(x)

The input is as shown above ie's RDD[Map[String, String]] Want to covert into dataframe with col1 and col2 as column names and foo and bar as one single row.

Upvotes: 1

Views: 552

Answers (1)

akuiper
akuiper

Reputation: 215117

You can create a case class, convert Maps in the rdd to case class and then toDF should work:

case class r(col1: Option[String], col2: Option[String])

RDD.map(m => r(m.get("col1"), m.get("col2"))).toDF.show
+----+----+
|col1|col2|
+----+----+
| foo| bar|
+----+----+

Upvotes: 2

Related Questions