CODEWITHSUNDEEP
apache-spark-sql
abiswas
abiswas

Reputation: 21

Spark SQL to explode array of structure

I have the below JSON structure which I am trying to convert to a structure with each element as column as shown below using Spark SQL. Explode(control) is not working. Can someone please suggest a way to do this?

Input:

{
"control" : [
    {
      "controlIndex": 0,
      "customerValue": 100.0,
      "guid": "abcd",
      "defaultValue": 50.0,
      "type": "discrete"
    },
    {
      "controlIndex": 1,
      "customerValue": 50.0,
      "guid": "pqrs",
      "defaultValue": 50.0,
      "type": "discrete"
    }
  ]
}

Desired output:

controlIndex customerValue guid defaultValult  type

0            100.0         abcd   50.0         discrete

1            50.0          pqrs   50.0         discrete

Upvotes: 2

Views: 13741

Answers (3)

Hao Tan
Hao Tan

Reputation: 1638

Addition to Paul Leclercq's answer, here is what can work.

import org.apache.spark.sql.functions.explode   
df.select(explode($"control").as("control")).select("control.*")

Upvotes: 3

Kunal Jha
Kunal Jha

Reputation: 7

Explode will not work here as its not a normal array column but an array of struct. You might want to try something like

df.select(col("control.controlIndex"), col("control.customerValue"), col ("control. guid"), col("control. defaultValue"), col(control. type))

Upvotes: -1

Paul Leclercq
Paul Leclercq

Reputation: 1018

Explode will create a new row for each element in the given array or map column

import org.apache.spark.sql.functions.explode   

df.select(
  explode($"control")
)

Upvotes: 1

Related Questions