CODEWITHSUNDEEP
scalaapache-sparkdataframe
nobody
nobody

Reputation: 8293

Fill scala column with nulls

I am getting the error Caused by: scala.MatchError: Null (of class scala.reflect.internal.Types$ClassNoArgsTypeRef) when I try to fill a DataFrame with null values to replace other values in it. How can I do this using Scala Spark 2.1?

Upvotes: 0

Views: 778

Answers (1)

zero323
zero323

Reputation: 330383

You can use isin and when. Required imports:

import org.apache.spark.sql.functions.when

Example data:

val toReplace = Seq("foo", "bar")

val df = Seq((1, "Jane"), (2, "foo"), (3, "John"), (4, "bar")).toDF("id", "name")

Query:

df.withColumn("name", when(!$"name".isin(toReplace: _*), $"name")).

and the result:

+---+----+
| id|name|
+---+----+
|  1|Jane|
|  2|null|
|  3|John|
|  4|null|
+---+----+

Upvotes: 4

Related Questions