CODEWITHSUNDEEP
scalaapache-spark
vikv
vikv

Reputation: 11

Get the column names from file in spark scala

I'm new to spark/scala. I have a file say config where I specify all the column names.

Config:

Id,
Emp_Name,
Dept,
Address,
Account

I have a dataframe where i select the column names like: 

df.select("id","Emp_Name","Dept","Address","Account").show()

Instead of specifying the column names in select, I want to get the column names from config file like

df.select(config-file_column_names).show()

Upvotes: 0

Views: 2265

Answers (1)

philantrovert
philantrovert

Reputation: 10082

You don't necessarily need the commas in your file if each column is in a different line.

This is the definition of select : 

def select(col: String, cols: String*): DataFrame
def select(cols: org.apache.spark.sql.Column*): DataFrame

We are going to use the second definition here.

import org.apache.spark.sql.functions.col

val colNames = sc.textFile("file").map(_.replaceAll(",", "") ).map(col(_)).collect

// Unpacking the array in `select`

df.select(colNames: _*).show

Upvotes: 1

Related Questions