CODEWITHSUNDEEP
scalaapache-sparkparquet
Easyhyum
Easyhyum

Reputation: 359

Spark save(write) parquet only one file

if i write

dataFrame.write.format("parquet").mode("append").save("temp.parquet")

in temp.parquet folder i got the same file numbers as the row numbers

i think i'm not fully understand about parquet but is it natural?

Upvotes: 34

Views: 61537

Answers (3)

Amar
Amar

Reputation: 278

You can set partitions as 1 to save as single file

dataFrame.repartition(1).write.format("parquet").mode("append").save("temp.parquet")

Upvotes: 21

bottaio
bottaio

Reputation: 5093

Although previous answers are correct you have to understand repercusions that come after repartitioning or coalescing to a single partition. All your data will have to be transferred to a single worker just to immediately write it to a single file.

As it is repeatidly mentioned throughout the internet, you should use repartition in this scenario despite the shuffle step that gets added to the execution plan. This step helps to use your cluster's power instead of sequentially merging files.

There is at least one alternative worth mentioning. You can write a simple script that would merge all the files into a single one. That way you will avoid generating massive network traffic to a single node of your cluster.

Upvotes: 7

y2k-shubham
y2k-shubham

Reputation: 11627

Use coalesce before write operation

dataFrame.coalesce(1).write.format("parquet").mode("append").save("temp.parquet")

EDIT-1

Upon a closer look, the docs do warn about coalesce

However, if you're doing a drastic coalesce, e.g. to numPartitions = 1, this may result in your computation taking place on fewer nodes than you like (e.g. one node in the case of numPartitions = 1)

Therefore as suggested by @Amar, it's better to use repartition

Upvotes: 33

Related Questions