CODEWITHSUNDEEP
scalaapache-sparkcase-class
anand babu
anand babu

Reputation: 364

how to create scala case class with struct types?

My dataframe schema looks like this, and it is created by defining a case class:

 |-- _id: struct (nullable = true)
 |    |-- oid: string (nullable = true)
 |-- message: string (nullable = true)
 |-- powerData: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- current: array (nullable = true)
 |    |    |    |-- element: double (containsNull = true)
 |    |    |-- delayStartTime: double (nullable = true)
 |    |    |-- idSub1: string (nullable = true)
 |    |    |-- motorNumber: integer (nullable = true)
 |    |    |-- power: array (nullable = true)
 |    |    |    |-- element: double (containsNull = true)

I created case class like this, but not sure how to declare StructFields in this case class.

case class CurrentSchema(_id: StructType, message: String, powerData: Array[StructType]

getting this error while apply schema against my DF.

val dfRef = MongoSpark.load[CurrentSchema](sparkSessionRef)

Exception in thread "main" scala.MatchError: org.apache.spark.sql.types.StructType (of class scala.reflect.internal.Types$ClassNoArgsTypeRef)

any one have done this like this? looking for some help.

Thanks in Advance.

Upvotes: 5

Views: 7561

Answers (1)

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41987

You will have to create separate case classes for each struct. 

case class IdStruct(old: String)
case class PdStruct(current: Array[Double], delayStartTime: Double, idSub1: String, motorNumber: Int, power: Array[Double])
case class CurrentSchema(_id: IdStruct, message: String, powerData: Array[PdStruct])

Upvotes: 15

Related Questions