CODEWITHSUNDEEP
dataframescalaapache-sparkdictionary
Haha
Haha

Reputation: 1009

Transform list of map to dataframe

I have the following data:

d = Some(List(Map(id -> 1, n -> Hi), Map(id -> 2, n -> Hello)))

I would like to transform it into a dataframe like the following:

+--------+
|id|n    |
+--------+
|1 |Hi   |
+--------+
|2 |Hello|
+--------+

I tried the following:

import spark.implicits._

val df = d
  .map( m => (m.get("id"),m.get("n")))
  .toDF("id", "n")

But im getting:

error: value get is not a member of Any
  .map( m => (m.get("id"),m.get("n")))

Upvotes: 0

Views: 62

Answers (1)

M_S
M_S

Reputation: 3733

Your top level here is Option and i think thats the reason why you cant handle it with single map. I managed to do it with something like this:

import spark.implicits._

val d = Some(List(Map("id" -> "1", "n" -> "Hi"), Map("id" -> "2", "n" -> "Hello")))

val data = d.fold (List.empty [(Option [String], Option [String])]) (_.map (m => (m.get ("id"), m.get ("n"))))

val df = data.toDF("id", "n").show()

Output:

+---+-----+
| id|    n|
+---+-----+
|  1|   Hi|
|  2|Hello|
+---+-----+

Upvotes: 2

Related Questions