CODEWITHSUNDEEP
pythondataframeapache-spark-sqlpyspark
picklesifen
picklesifen

Reputation: 43

Pyspark dataframe column contains array of dictionaries, want to make each key from dictionary into a column

I currently have a dataframe like this:

+-------+-------+-------+-------+
| Id    |value_list_of_dicts    |
+-------+-------+-------+-------+
| 1     |[{"val1":0, "val2":0}, |
|       |{"val1":2, "val2":5}]  |
+-------+-------+-------+-------+
| 2     |[{"val1":9, "val2":10},|
|       |{"val1":1, "val2":2}]  |
+-------+-------+-------+-------+

Each list contains exactly 30 dictionaries, and the values may differ but the key names are always the same. I want my dataframe to look like this:

+-------+-------+-------+
| Id    |val1   |val2   |
+-------+-------+-------+
| 1     | 0     | 0     |
+-------+-------+-------+
| 1     | 2     | 5     |
+-------+-------+-------+
| 2     | 9     | 10    |
+-------+-------+-------+
| 2     | 1     | 2     |
+-------+-------+-------+

What is the best way to do this?

Upvotes: 4

Views: 4391

Answers (1)

Addy
Addy

Reputation: 427

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

from pyspark.sql.types import *
from datetime import datetime
from pyspark.sql import *
from collections import *
from pyspark.sql.functions import udf,explode
from pyspark.sql.types import StringType
df= spark.createDataFrame(
    [
        (1, [{"val1":0, "val2":0},{"val1":2, "val2":5}]), 
        (2, [{"val1":9, "val2":10},{"val1":1, "val2":2}])
       
        ],("ID","List")
    )

df2 = df.select(df.ID,explode(df.List).alias("Column1") )
df2.withColumn("Val1", F.col("Column1").getItem("val1")).withColumn("Val2", F.col("Column1").getItem("val2")).show(truncate=False)

Output:

+---+-----------------------+----+----+
|ID |Column1                |Val1|Val2|
+---+-----------------------+----+----+
|1  |[val2 -> 0, val1 -> 0] |0   |0   |
|1  |[val2 -> 5, val1 -> 2] |2   |5   |
|2  |[val2 -> 10, val1 -> 9]|9   |10  |
|2  |[val2 -> 2, val1 -> 1] |1   |2   |

+---+-----------------------+----+----+

Upvotes: 2

Related Questions