How to filter the null data from Cassandra in Spark?

Question

I want to filter the Null value which selected from Cassandra.

Here is my query:

scala> var rdd =  sc.cassandraTable("keyspace", "table").select("month", "timetag", "name").where("month = ?", "201704")
scala> var data = rdd.filter(_.getString("name") != null)

If I use getString("xxx") compare to null, it will show

"java.lang.NullPointerException: Unexpected null value of column 2. Use get[Option[...]] to receive null values."

After that, I tried to use getStringOption to compare.

scala> var rdd =  sc.cassandraTable("keyspace", "table").select("month", "timetag", "name").where("month = ?", "201704")
scala> var data = rdd.filter(_.getStringOption("name") != null)

This time, it didn't show any error message. But the data is not filtered. The null data is still there.

Does anyone know what is different between getString and getStringOption? Or is there any other way I can compare the data from Cassandra is null?

Thanks a lot!

Answer 1

Try doing:

_.getStringOption("name").isDefined

instead of:

 _.getStringOption("name") != null

Update

It might be relevant to expand a little on the Option part.

The getStringOption (which is internally implemented as get[Option[String]] returns a scala Option class.

Option is an idiomatic way in functional programming to represent something which either has some value (denoted as Some(value)) or no value (denoted as None).

Unlike the use of null in java, None is actually an object with various functionality (the example here is isDefined). This makes it easy to combine multiple operation on object while making sure that if something on the way is no defined then we would not get a null pointer exception.

For more information on the Option type see the scala class documentation or this stack overflow answer or this stack overflow answer

Answer 2

Use getOption variants when you expect to receive Cassandra null values

_.get[Option[String]]("name")
res8: Option[String] = Some(abc)