CODEWITHSUNDEEP
export-to-csvpyspark
Mus
Mus

Reputation: 303

Save Pyspark Dataframe into csv without headers

I'm looking for a way to save a dataframe into a csv file without specifying headers.

I tried the code below but it did not work.

ratingsDF.coalesce(1).write.option("header", "false")\
.csv("csv_file_without_headers.csv")

For the dataframe below:

+---------+-----+
|   x|   y|    z|
+----+----+-----+
|   0|   a|    5|
|   1|   b|   12|
|   2|   c|    7|
|   3|   d|   27|
|   4|   e|  149|
|   5|   f|   19|
+---------+-----+

The expected result of csv:

O,a,5
1,b,12
2,c,7
3,d,27
4,e,149
5,f,19

Upvotes: 2

Views: 3922

Answers (3)

adarsh vemali
adarsh vemali

Reputation: 1

You can simply do:

ratingsDF.coalesce(1).write.csv("csv_file_without_headers.csv", header = False)

Upvotes: 0

Lamanus
Lamanus

Reputation: 13581

Try this,

ratingsDF.coalesce(1).write.csv("/path/to/save/csv/")

where it will save the csv without header by default. You cannot specify the csv file name but the path only.

Upvotes: 1

notNull
notNull

Reputation: 31540

Your option looks correct and csv files that is getting written will not be having headers.

In Spark it is not possible to write to file csv_file_without_headers.csv instead check for csv_file_without_headers.csv directory.

  • In the directory you can see all the files in the directory with out header.

Upvotes: 2

Related Questions