CODEWITHSUNDEEP
scalaapache-spark
neuroh
neuroh

Reputation: 377

Spark Select with a List of Columns Scala

I am trying to find a good way of doing a spark select with a List[Column, I am exploding a column than passing back all the columns I am interested in with my exploded column.

var columns = getColumns(x) // Returns a List[Column]
tempDf.select(columns)   //trying to get

Trying to find a good way of doing this I know, if it were a string I could do something like

val result = dataframe.select(columnNames.head, columnNames.tail: _*)

Upvotes: 14

Views: 47213

Answers (1)

Franzi
Franzi

Reputation: 1871

For spark 2.0 seems that you have two options. Both depends on how you manage your columns (Strings or Columns).

Spark code (spark-sql_2.11/org/apache/spark/sql/Dataset.scala):

def select(cols: Column*): DataFrame = withPlan {
  Project(cols.map(_.named), logicalPlan)
}

def select(col: String, cols: String*): DataFrame = select((col +: cols).map(Column(_)) : _*)

You can see how internally spark is converting your head & tail to a list of Columns to call again Select.

So, in that case if you want a clear code I will recommend:

If columns: List[String]:

import org.apache.spark.sql.functions.col
df.select(columns.map(col): _*)

Otherwise, if columns: List[Columns]:

df.select(columns: _*)

Upvotes: 40

Related Questions