StackOverflow Questions for Tag: spark-shuffle

figs_and_nuts

Reputation: 5771

Understanding the shuffle in spark

apache-sparkpysparkspark-shuffle

Score: 4

Answers: 1

Junbo Du

Reputation: 1

How exactly do partitions work in Spark shuffling?

apache-sparkdata-processinginternalsspark-shuffle

Score: 0

Answers: 0

bmcristi

Reputation: 103

Spark window operation with data skew

apache-sparkpysparkspark-shuffle

Score: 0

Answers: 1

Tom

Reputation: 6342

Does Spark Dynamic Allocation depend on external shuffle service to work well?

apache-sparkspark-shuffle

Score: 4

Answers: 2

Lakesh Kumar

Reputation: 1

Migration from Spark 3.1 to 3.3 org.apache.spark.shuffle.FetchFailedException failures

apache-sparkdataprocspark3spark-shuffle

Score: 0

Answers: 1

Denziloe

Reputation: 8152

Does Spark shuffle write all intermediate data to disk?

apache-sparkpysparkspark-shuffle

Score: 3

Answers: 3

Mandroid

Reputation: 7524

How wide transformations are influenced by shuffle partition config

apache-sparkapache-spark-datasetspark-shuffle

Score: 0

Answers: 1

Josh Chang

Reputation: 13

Count words from a list within array columns without invoking a shuffle

arraysapache-sparkpysparkapache-spark-sqlspark-shuffle

Score: 1

Answers: 3

atul gurale

Reputation: 39

No space left on device error in Spark Scala

scalaapache-sparkapache-spark-sqlspark-shuffle

Score: 0

Answers: 0

湘晗刚

Reputation: 11

org.apache.spark.shuffle.FetchFailedException: The relative remote executor is dead

apache-sparkspark-shuffle

Score: 1

Answers: 0

spark-noob

Reputation: 31

HashPartioning dataframes to achieve co-partitioning during join in PySpark

apache-sparkpysparkapache-spark-sqlpartitioningspark-shuffle

Score: 3

Answers: 0

figs_and_nuts

Reputation: 5771

How to avoid unnecessary shuffle in pyspark?

apache-sparkpysparkapache-spark-sqlspark-shuffle

Score: 2

Answers: 1

user3834294

Reputation: 65

filter data in tfrecord with spark/scala without aggregate steps?

tensorflowapache-sparktfrecordspark-shuffle

Score: 2

Answers: 0

figs_and_nuts

Reputation: 5771

How does spark calculate the number of reducers in a hash shuffle?

apache-sparkapache-spark-sqlmapreducespark-shuffle

Score: 2

Answers: 1

figs_and_nuts

Reputation: 5771

What is spark spill (disk and memory both)?

apache-sparkpysparkapache-spark-sqlspark-uispark-shuffle

Score: 5

Answers: 1

Rushabh Gujarathi

Reputation: 146

What is the difference between spark.shuffle.partition and spark.repartition in spark?

apache-sparkbigdataapache-spark-sql-repartitionspark-shuffle

Score: 3

Answers: 1

evalgor

Reputation: 3

Repartition on non-deterministic expression

apache-sparkapache-spark-sqlspark-shuffle

Score: 0

Answers: 1

Thomas Decaux

Reputation: 22691

Spark shuffle service on local shared dir with Ceph on kubernetes

apache-sparkspark-shuffle

Score: 1

Answers: 0

Garret Wilson

Reputation: 21466

Spark NullPointerException: Cannot invoke invalidateSerializedMapOutputStatusCache() because "shuffleStatus" is null

javaapache-sparkspark-shuffle

Score: 0

Answers: 2

Surendiran Balasubramanian

Reputation: 25

how to decide number of executors for 1 billion rows in spark

apache-sparkpysparkspark-shuffle

Score: 0

Answers: 1

PreviousPage 1Next

StackOverflow Questions for Tag: spark-shuffle

Understanding the shuffle in spark

How exactly do partitions work in Spark shuffling?

Spark window operation with data skew

Does Spark Dynamic Allocation depend on external shuffle service to work well?

Migration from Spark 3.1 to 3.3 org.apache.spark.shuffle.FetchFailedException failures

Does Spark shuffle write all intermediate data to disk?

How wide transformations are influenced by shuffle partition config

Count words from a list within array columns without invoking a shuffle

No space left on device error in Spark Scala

org.apache.spark.shuffle.FetchFailedException: The relative remote executor is dead

HashPartioning dataframes to achieve co-partitioning during join in PySpark

How to avoid unnecessary shuffle in pyspark?

filter data in tfrecord with spark/scala without aggregate steps?

How does spark calculate the number of reducers in a hash shuffle?

What is spark spill (disk and memory both)?

What is the difference between spark.shuffle.partition and spark.repartition in spark?

Repartition on non-deterministic expression

Spark shuffle service on local shared dir with Ceph on kubernetes

Spark NullPointerException: Cannot invoke invalidateSerializedMapOutputStatusCache() because &quot;shuffleStatus&quot; is null

how to decide number of executors for 1 billion rows in spark

Spark NullPointerException: Cannot invoke invalidateSerializedMapOutputStatusCache() because "shuffleStatus" is null