CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Ramesh
Ramesh

Reputation: 121

Fetching columns dynamically from dataframe , column name would come in variable

I am not able to fetch values for given dynamic columns. Any help ?

var dynamicColumns = "col(\"one\"),col(\"two\"),col(\"three\")"

dataFrame.select(dynamicColumns)

Upvotes: 0

Views: 243

Answers (1)

Alper t. Turker
Alper t. Turker

Reputation: 35249

Just use names alone:

val dynamicColumns = Seq("one", "two", "three")
dataFrame.select(dynamicColumns map col: _*)

and if you don't have control over the format, use regexp to extract names first

val dynamicColumns = "col(\"one\"),col(\"two\"),col(\"three\")"
val p = """(?<=col\(").+?(?="\))""".r

dataFrame.select(p.findAllIn(dynamicColumns) map col toSeq: _*)

Upvotes: 1

Related Questions