CODEWITHSUNDEEP
apache-sparkshuffle
S.A
S.A

Reputation: 93

How to enable tungsten sort shuffle in Spark 2.1?

In previous versions, there is a configuration called spark.shuffle.manage which is used to determine the type of shuffle algorithm in Spark. Since Spark 2.0, this configuration is deleted. The default shuffle algorithm is the sort-based. As my understanding, the Tungsten shuffle will be enabled only if all the requirements are satisfied. How can I know whether the current job uses the original sort-based shuffle or Tungsten sort?

Thank you very much.

Upvotes: 2

Views: 2049

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74729

SortShuffleManager is the one and only ShuffleManager in Apache Spark.

In other words, there's no way you could use any other ShuffleManager but SortShuffleManager (unless you enabled one using spark.shuffle.manager property).

Upvotes: 3

Related Questions