CODEWITHSUNDEEP
scalaapache-spark
Alexander Lopatin
Alexander Lopatin

Reputation: 582

Spark: convert array of arrays to array of strings?

I have column with type Array of Arrays I need to get column array of string.

+--------------------------+
|field                     |
+--------------------------+
|[[1, 2, 3], [1, 2, 3], []]|
+--------------------------+

I need to get:

+--------------------------+
|field                     |
+--------------------------+
|["123", "123", ""]        |
+--------------------------+

Can I do this in Spark without using UDF?

Upvotes: 0

Views: 95

Answers (1)

Mohana B C
Mohana B C

Reputation: 5487

You can use transform higher order function,

import spark.implicits._

val df = Seq(Seq(Seq(1,2,3), Seq(1,2,3), Seq())).toDF("field")

df.withColumn("field", expr("transform(field, v->concat_ws('',v))"))
    .show

+------------+
|       field|
+------------+
|[123, 123, ]|
+------------+

Upvotes: 2

Related Questions