CODEWITHSUNDEEP
scalahadoopapache-sparkhive
Darshan
Darshan

Reputation: 81

Null check for Double/Int Value in Spark

I am new in Spark, How can I check for for Null value in Double and Int value in scala or Spark.

Like for String We can do like this :

val value = (FirstString.isEmpty()) match {
          case true => SecondString
          case _    => FirstString
        }

I searched for it a lot but i found only for String value. Can you please suggest me for other datatype as well.

Thanks in advance.

Upvotes: 3

Views: 15039

Answers (3)

Alexey Romanov
Alexey Romanov

Reputation: 170723

isEmpty is not at all the same as "check for null". Calling isEmpty on null will fail:

val s: String = null
s.isEmpty // throws NullPointerException

Int and Double can't be null (neither can any other primitive types), so there is no need to check if they are. If you are talking specifically about Spark Rows, you need to check for null before getting an Int/Double/other primitive value:

It is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null.

Upvotes: 3

Samar
Samar

Reputation: 2101

null is only applicable for AnyRef (i.e non primitive types) types in Scala. AnyVal types can not be set to null.

For example:

// the below are AnyVal(s) and wont compile
val c: Char = null
val i: Int = null
val d: Double = null

String is an AnyRef and so can be null:

// this is ok!
val c: String = null

That's why pattern matching nulls to Int/Double types is not possible:

// wont compile!
null match {
        case a:Int => "is a null Int"
        case _ => "something else"
        }

Upvotes: 6

codejitsu
codejitsu

Reputation: 3182

May be you can simply use Options. So like:

val d: Double = ...

val isNull = Option(d).isDefined

Or you can use pattern matching:

val d: Double = ...

Option(d) match {
  case Some(v) => use v
  case _ => you got Null
}

Upvotes: 4

Related Questions