CODEWITHSUNDEEP
scalaapache-spark
user1615666
user1615666

Reputation: 3451

spark dataframe trim column and convert

In Scala / Spark, how to convert empty string, like " ", to "NULL" ? need to trim it first and then convert to "NULL". Thanks.

dataframe.na.replace("cut", Map(" " -> "NULL")).show //wrong

Upvotes: 8

Views: 22967

Answers (2)

Kapil
Kapil

Reputation: 75

Please use the package below to resolve issue

import org.apache.spark.sql.functions.trim

Upvotes: 0

zero323
zero323

Reputation: 330093

You can create a simple function to do it. First a couple of imports: 

import org.apache.spark.sql.functions.{trim, length, when}
import org.apache.spark.sql.Column

and the definition:

def emptyToNull(c: Column) = when(length(trim(c)) > 0, c)

Finally a quick test:

val df = Seq(" ", "foo", "", "bar").toDF
df.withColumn("value", emptyToNull($"value"))

which should yield following result:

+-----+
|value|
+-----+
| null|
|  foo|
| null|
|  bar|
+-----+

If you want to replace empty string with string "NULL you can add otherwise clause:

def emptyToNullString(c: Column) = when(length(trim(c)) > 0, c).otherwise("NULL")

Upvotes: 9

Related Questions