CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
am1212
am1212

Reputation: 553

How to make a dataframe into a case class?

Lots of documentations show that it's possible to go from a case class to a dataframe, but I haven't been able to find a good way of going from a dataframe to a case class.

Let's say I have a dataframe with 50 columns, but would like to select out about 5 columns and make it into a new table. I could approach it this way:

sqlContext.sql("select [1, 2, 3, 4, 5] from test").registerTempTable("newTable")

But the newTable will have some other columns like 6, 7 as a customized value (or 0 for now, but this column just doesn't exist in the test table). To solve this, I tried to create a case class that looks like this:

case class newTable(1, 2, 3, 4, 5, 6, 7)

In the end, I would want to extract column 1 through 5 from the test table, then input 6, 7 whatever I would like to. I just haven't found a good way of doing this.

Upvotes: 1

Views: 1214

Answers (1)

HuntsMan
HuntsMan

Reputation: 792

you can use like this:

dataframe.select($"1".as("1"), $"2".as("2"), $"3".as("3"), $"4".as("4"), $"5".as("5")).as[newTable]

Note : you should match the column name as the field name in your case class

Upvotes: 2

Related Questions