CODEWITHSUNDEEP
apache-spark
Funzo
Funzo

Reputation: 1290

Convert spark row to case class and fail if value is null

I want to convert a Dataframe to Dataset[CC]. 

case class CC(id: String, value: String)
df.as[CC]

However if in the id or value is null I would like the operation to throw an exception. Is this possible?

Upvotes: 0

Views: 161

Answers (1)

chlebek
chlebek

Reputation: 2451

maybe this way:

 val ds = df.map(s=>{
    s.toSeq.foreach(x=>{
      if (x == null) throw new Exception("null value: "+s.mkString(","))
    })
    new CC(s.getString(0),s.getString(1))
  })

    +----+----+
    | _c0| _c1|
    +----+----+
    |   1|   a|
    |   2|null|
    |   3|   b|
    |null|null|
    |   5|null|
    +----+----+

and exception:

`Caused by: java.lang.Exception: null value: 2,null`

Upvotes: 1

Related Questions