CODEWITHSUNDEEP
scalaapache-sparkdataset
Juncheng ZHOU
Juncheng ZHOU

Reputation: 125

How to set the type of array with dataset in spark scala

I have a source data like this:

{A:123,B:"Hello world",C:[{D:123,E:"Spark"}]}

And i have a Object:

case class TestClass (A:Int;B:String;C:???)
val obj:Dataset[TestClass] = df.as[TestClass]

How should I define the type of C?

Upvotes: 2

Views: 493

Answers (1)

user8784925
user8784925

Reputation: 46

One option

case class Nested(D: Long, E: String)
case class TestClass (A: Long, B:String, C: Seq[Nested])

Usage:

spark.read.json(sc.parallelize(
  Seq("""{"A": 123, "B": "Hello world", "C": [{"D": 123, "E": "Spark"}]}"""
))).as[TestClass].show

+---+-----------+-------------+
|  A|          B|            C|
+---+-----------+-------------+
|123|Hello world|[[123,Spark]]|
+---+-----------+-------------+

Upvotes: 3

Related Questions