CODEWITHSUNDEEP
dataframecsvapache-sparkpysparkapache-spark-sql
frenchast
frenchast

Reputation: 33

Pyspark read csv

After read the csv, I'm getting an unexpected output:

MessageName;ContactKey;DateTimeSend;MessageContent;MessageOpened;OpenDate;TimeInApp;Platform;PlatformVersion;Status
20200903 - NL SPAARUPDATE Augustus;0031t00000A4w0xAAB;09/03/2020 8:09;Vorige maand heb je dankzij de Lidl-Plus app %%savings%% euro gespaard. Goed bezig! ??????;no;;;iPhone OS;12.4.5;Success

As you can imagine, the output required it's the split of this information into columns and cells to create a normal dataframe.

I tried the following code:

df = spark.read.csv('/FileStore/tables/BE_August_monthlysaving.csv', header='true')

display(df)

Alternatively, I tried to use , delimiter=';' before and after the header but when I do this I get the following error:

csv() got an unexpected keyword argument 'delimiter'

Any idea how to solve this output?

Upvotes: 0

Views: 1156

Answers (1)

mck
mck

Reputation: 42412

Use sep instead of delimiter:

df = spark.read.csv('/FileStore/tables/BE_August_monthlysaving.csv', header='true', sep=';')

Or you can put it as an option:

df = spark.read.option('delimiter', ';').csv('/FileStore/tables/BE_August_monthlysaving.csv', header='true')

Upvotes: 5

Related Questions