CODEWITHSUNDEEP
scalaapache-spark
lserlohn
lserlohn

Reputation: 6206

How to explode an array into multiple columns in Spark

I have a spark dataframe looks like:

id   DataArray
a    array(3,2,1)
b    array(4,2,1)     
c    array(8,6,1)
d    array(8,2,4)

I want to transform this dataframe into:

id  col1  col2  col3
a    3     2     1
b    4     2     1
c    8     6     1 
d    8     2     4

What function should I use?

Upvotes: 11

Views: 17623

Answers (2)

user9554572
user9554572

Reputation: 236

Use apply:

import org.apache.spark.sql.functions.col

df.select(
  col("id") +: (0 until 3).map(i => col("DataArray")(i).alias(s"col$i")): _*
)

Upvotes: 22

koiralo
koiralo

Reputation: 23099

You can use foldLeft to add each columnn fron DataArray

make a list of column names that you want to add 

val columns = List("col1", "col2", "col3")

columns.zipWithIndex.foldLeft(df) {
  (memodDF, column) => {
    memodDF.withColumn(column._1, col("dataArray")(column._2))
  }
}
  .drop("DataArray")

Hope this helps!

Upvotes: 4

Related Questions