CODEWITHSUNDEEP
csvapache-sparkpysparkapache-spark-sql
Antoine Pietri
Antoine Pietri

Reputation: 813

Spark: write a CSV with null values as empty columns

I'm using PySpark to write a dataframe to a CSV file like this:

df.write.csv(PATH, nullValue='')

There is a column in that dataframe of type string. Some of the values are null. These null values display like this:

...,"",...

I would like them to be display like this instead:

...,,...

Is this possible with an option in csv.write()?

Thanks!

Upvotes: 12

Views: 25379

Answers (1)

Kafels
Kafels

Reputation: 4059

Easily with emptyValue option setted

emptyValue: sets the string representation of an empty value. If None is set, it use the default value, "".

from pyspark import Row
from pyspark.shell import spark

df = spark.createDataFrame([
    Row(col_1=None, col_2='20151231', col_3='Hello'),
    Row(col_1=2, col_2='20160101', col_3=None),
    Row(col_1=3, col_2=None, col_3='World')
])

df.write.csv(PATH, header=True, emptyValue='')

Output

col_1,col_2,col_3
,20151231,Hello
2,20160101,
3,,World

Upvotes: 20

Related Questions