CODEWITHSUNDEEP
dataframescalaapache-sparkfoldleft
Kylo
Kylo

Reputation: 109

scala spark foldLeft on map with array

I have a configuration in a form of a map:

val config =  Map[String, Array[String]] = Map("column1" -> Array("field1"), "column2" -> Array("field1","field2"))

I want to use foldLeft to apply this to a dataframe and dynamically filter nested fields using dropFields functions.

I came out with:

val result = config.foldLeft(""){(k, v) =>
  df.withColumn( v._1, col(k + v._1).dropFields(v._2: _*))
}

but struggle to make foldleft work, any help would be appreciated.

Upvotes: 0

Views: 287

Answers (0)

Related Questions