CODEWITHSUNDEEP
apache-sparkdataframe
G G
G G

Reputation: 1069

Spark DataFrame making column null value to empty

I have joined two data frames with left outer join. Resulting data frame has null values. How do to make them as empty instead of null.

| id|quantity|
+---+--------
|  1|    null|
|  2|    null|
|  3|    0.04

And here is the schema

root
|-- id: integer (nullable = false)
|-- quantity: double (nullable = true)

expected output

| id|quantity|
+---+--------
|  1|        |
|  2|        |
|  3|    0.04

Upvotes: 2

Views: 3745

Answers (1)

TheMP
TheMP

Reputation: 8427

You cannot make them "empty", since they are double values and empty string "" is a String. The best you can do is leave them as nulls or set them to 0 using fill function:

val df2 = df.na.fill(0.,Seq("quantity"))

Otherwise, if you really want to have empty quantities, you should consider changing quantity column type to String.

Upvotes: 4

Related Questions