CODEWITHSUNDEEP
apache-sparkapache-spark-sql
user7687835
user7687835

Reputation: 79

Convert List[Map<String,String>] to spark dataframe

I want to convert List[Map] to a spark dataframe,the keys of Map are sname,the keys of Map are DataFrame's columns

Upvotes: 0

Views: 2633

Answers (2)

Mohamed_Raed
Mohamed_Raed

Reputation: 31

If you already have res which is a List[Map[String,String]] :

res: List[Map[String,String]] = List(Map(A -> a1, B -> b1, C -> c1), Map(A -> a2, B -> b2, C -> c2))

You can do this to create you dataframe:

//create your rows
val rows = res.map(m => Row(m.values.toSeq:_*))

//create the schema from the header
val header = res.head.keys.toList
val schema = StructType(header.map(fieldName => StructField(fieldName, StringType, true)))

//create your rdd
val rdd = sc.parallelize(rows)

//create your dataframe using 
val df = spark.createDataFrame(rdd, schema)

You can output the result with df.show() :

+---+---+---+
|  A|  B|  C|
+---+---+---+
| a1| b1| c1|
| a2| b2| c2|
+---+---+---+

Note that you can also create your schema in this way:

val schema = StructType(
   List(
     StructField("A", StringType, true),
     StructField("B", StringType, true),
     StructField("C", StringType, true)
   )
 )

Upvotes: 2

H Roy
H Roy

Reputation: 635

Here you can do this way

val map1 = {"a"->1}
val map2 = {"b"->2}
val lst = List(map1,map2)
val lstDF = lst.toDF
lstDF.take(2).foreach(println)

Upvotes: 0

Related Questions