CODEWITHSUNDEEP
apache-sparkapache-spark-sql
vishalraj
vishalraj

Reputation: 115

Can we reorder spark dataframe's columns?

I am creating dataframe as per given schema, after that i want to create new dataframe by reordering the existing dataframe. Can it be possible the re-ordering of columns in spark dataframe?

object Demo extends Context {

  def main(args: Array[String]): Unit = {
    val emp = Seq((1,"Smith",-1,"2018","10","M",3000),
      (2,"Rose",1,"2010","20","M",4000),
      (3,"Williams",1,"2010","10","M",1000),
      (4,"Jones",2,"2005","10","F",2000),
      (5,"Brown",2,"2010","40","",-1),
      (6,"Brown",2,"2010","50","",-1)
    )
    val empColumns = Seq("emp_id","name","superior_emp_id","year_joined",
      "emp_dept_id","gender","salary")
    import sparkSession.sqlContext.implicits._
    val empDF = emp.toDF(empColumns: _*)
    empDF.show(false)
  }
}

Current DF:
+------+--------+---------------+-----------+-----------+------+------+
|emp_id|name    |superior_emp_id|year_joined|emp_dept_id|gender|salary|
+------+--------+---------------+-----------+-----------+------+------+
|1     |Smith   |-1             |2018       |10         |M     |3000  |
|2     |Rose    |1              |2010       |20         |M     |4000  |
|3     |Williams|1              |2010       |10         |M     |1000  |
|4     |Jones   |2              |2005       |10         |F     |2000  |
|5     |Brown   |2              |2010       |40         |      |-1    |
|6     |Brown   |2              |2010       |50         |      |-1    |
+------+--------+---------------+-----------+-----------+------+------+

I want output as this following df, where gender and salary column re-ordered
New DF:
+------+--------+------+------+---------------+-----------+-----------+
|emp_id|name    |gender|salary|superior_emp_id|year_joined|emp_dept_id|
+------+--------+------+------+---------------+-----------+-----------+
|1     |Smith   |M     |3000  |-1             |2018       |10         |
|2     |Rose    |M     |4000  |1              |2010       |20         |
|3     |Williams|M     |1000  |1              |2010       |10         |
|4     |Jones   |F     |2000  |2              |2005       |10         |
|5     |Brown   |      |-1    |2              |2010       |40         |
|6     |Brown   |      |-1    |2              |2010       |50         |
+------+--------+------+------+---------------+-----------+-----------+

Upvotes: 0

Views: 3467

Answers (2)

QuickSilver
QuickSilver

Reputation: 4045

Scala way of doing it

//Order the column names as you want
    val columns = Array("emp_id","name","gender","salary","superior_emp_id","year_joined","emp_dept_id")
        .map(col)

    //Pass it to select
    df.select(columns: _*)

Upvotes: 0

Ala Tarighati
Ala Tarighati

Reputation: 3817

Just use select() to re-order the columns:

df = df.select('emp_id','name','gender','salary','superior_emp_id','year_joined','emp_dept_id')

It will be shown according to your ordering in select() argument.

Upvotes: 3

Related Questions