CODEWITHSUNDEEP
apache-sparkpysparkrddconcept
Tim
Tim

Reputation: 1069

Spark cartesian doesn't cause shuffle?

So, I tried to test on Spark operations that cause shuffling based on this stackoverflow post: LINK. However, it doesn't make sense for me when the cartesian operation doesn't cause shuffling in Spark since they need to move the partitions across the network in order to put them together locally.

How does Spark actually do its cartesian and distinct operations behind the scene??

Upvotes: 3

Views: 514

Answers (1)

zero323
zero323

Reputation: 330093

Shuffle is an operation which is specific to RDDs of key-value pairs (RDD[(T, U)] commonly described as PairRDDs or PairwiseRDDs) and is more or less equivalent to shuffle phase in Hadoop. A goal of shuffle is to move data to specific executor based on key value and Partitioner.

There are different types of operations in Spark, which require network traffic, but don't use the same type of logic as shuffle and not always require key-value pairs. Cartesian product is one of these operations. It moves data between machines (in fact it causes much more expensive data movements) but doesn't establish relationship between keys and executors.

Upvotes: 2

Related Questions