CODEWITHSUNDEEP
scalaapache-sparkdataframeapache-spark-sql
Sorabh Kumar
Sorabh Kumar

Reputation: 176

How to create a dataframe from a list of Scala custom objects

We can create a dataframe from a list of Java objects using:

DataFrame df = sqlContext.createDataFrame(list, Example.class);

In case of Java, Spark can infer the schema directly from the class, in this case Example.class.

Is there a way to do the same in case of Scala?

Upvotes: 0

Views: 3101

Answers (1)

Raphael Roth
Raphael Roth

Reputation: 27373

if you use case classes in scala, this works out of the box 

// define this class outside main method
case class MyCustomObject(id:Long,name:String,age:Int)

import spark.implicits._

val df = Seq(
  MyCustomObject(1L,"Peter",34),
  MyCustomObject(2L,"John",52)
).toDF()

df.show()

+---+-----+---+
| id| name|age|
+---+-----+---+
|  1|Peter| 34|
|  2| John| 52|
+---+-----+---+

If you want to use a non-case class, you need to extend the trait Product and implement these methods yourself

Upvotes: 6

Related Questions