CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
arya
arya

Reputation: 476

unable to getoutput for spark case classes

i am trying to implement

using spark 2.4.8 and sbt version 1.4.3 using intellij

code:

val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._

case class Person(id:Int,Name:String,cityId:Long)
case class City(id:Long,Name:String)

val family=Seq(Person(1,"john",11),(2,"MAR",12),(3,"Iweta",10)).toDF
val cities=Seq(City(11,"boston"),(12,"dallas")).toDF


error:
Exception in thread "main" java.lang.NoClassDefFoundError: no Java class corresponding to Product with Serializable found
    at scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1300)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:192)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:54)
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:60)
    at org.apache.spark.sql.Encoders$.product(Encoders.scala:275)
    at org.apache.spark.sql.LowPrioritySQLImplicits$class.newProductEncoder(SQLImplicits.scala:248)
    at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:34)
    at usingcaseclass$.main(usingcaseclass.scala:26)
    at usingcaseclass.main(usingcaseclass.scala)

  case class Salary(depName: String, empNo: Long, salary: Long)
val empsalary = Seq(Salary("sales", 1, 5000), Salary("personnel", 2, 3900)).toDS
empsalary.show(false)

value toDS is not a member of Seq[Salary]
val empsalary = Seq(Salary("sales", 1, 5000), Salary("personnel", 2, 3900)).toDS

any idea how to prevent this error

Upvotes: 0

Views: 451

Answers (1)

Mohana B C
Mohana B C

Reputation: 5487

You have defined Seq in a wrong way, which will result in Seq[Product with Serializable] not Seq[T] on which toDF works.

Below modified lines should work for you.

val family=Seq(Person(1,"john",11),Person(2,"MAR",12),Person(3,"Iweta",10))

family.toDF().show()

+---+-----+------+
| id| Name|cityId|
+---+-----+------+
|  1| john|    11|
|  2|  MAR|    12|
|  3|Iweta|    10|
+---+-----+------+

Upvotes: 2

Related Questions