CODEWITHSUNDEEP
scalaapache-spark
Nurzhan Nogerbek
Nurzhan Nogerbek

Reputation: 5236

How in Spark to set default value if column has NULL?

I am tring to join 2 DataFrames in Spark Framework.

val final = mf1.join(mf2,mf1("KEY")===mf2("MOBILE_KEY"), "left").select(
        mf1("*"),
        mf2("MOBILE_TYPE")
    )
)

Next code works but return null value in MOBILE_TYPE column to some records. How set default value if column has null value?

Upvotes: 1

Views: 6563

Answers (1)

undefined_variable
undefined_variable

Reputation: 6218

val final = mf1.join(mf2,mf1("KEY")===mf2("MOBILE_KEY"), "left").select(
        mf1("*"),
        mf2("MOBILE_TYPE")
    )
).withColumn("MOBILE_TYPE",when(col("MOBILE_TYPE").isNull,"DEFAULT_VALUE").otherwise(col("MOBILE_TYPE")))

withColumn and when API can work.

Upvotes: 2

Related Questions