CODEWITHSUNDEEP
javaapache-spark
VB_
VB_

Reputation: 45692

Spark partitions size on coalesce

Is there any way to ask Spark to make partitions of equal size on coalesce?

Stats[] stats = sparkSession
    .read()
    .parquet(salesPath)
    .coalesce(5) // may produce partitions of 120, 1, 1, 0, 0

Upvotes: 2

Views: 1378

Answers (1)

Erni Durdevic
Erni Durdevic

Reputation: 118

There is no way to have equal size partitions with coalesce. You should use repartition for that.

Check out https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html#transformations

Upvotes: 2

Related Questions