CODEWITHSUNDEEP
scalaapache-spark
René Kolařík
René Kolařík

Reputation: 1284

How to distribute data to worker nodes

I have a general question regarding Apache Spark and how to distribute data from driver to executors. I load a file with 'scala.io.Source' into collection. Then I parallelize the collection with 'SparkContext.parallelize'. Here begins the issue - when I don't specify the number of partitions, then the number of workers is used as the partitions value, task is sent to nodes and I got the warning that recommended task size is 100kB and my task size is e.g. 15MB (60MB file / 4 nodes). The computation then ends with 'OutOfMemory' exception on nodes. When I parallelize to more partitions (e.g. 600 partitions - to get the 100kB per task). The computations are performed successfully on workers but the 'OutOfMemory' exceptions is raised after some time in the driver. This case, I can open spark UI and observe how te memory of driver is slowly consumed during the computation. It looks like the driver holds everything in memory and doesn't store the intermediate results on disk.

My questions are:

Thanks

Upvotes: 3

Views: 2312

Answers (1)

maasg
maasg

Reputation: 37435

  • How to distribute data 'the right way'?

    You will need a distributed file system, such as HDFS, to host your file. That way, each worker can read a piece of the file in parallel. This will deliver better performance than serializing and the data.

  • How to prevent memory exceptions?

    Hard to say without looking at the code. Most operations will spill to disk. If I had to guess, I'd say you are using groupByKey ?

  • Into how many partitions to divide RDD?

    I think the rule of thumbs (for optimal parallelism) is 2-4x the amount of cores available for your job. As you have done, you can compromise time for memory usage.

  • Is there a way how to tell driver/worker to swap? Is it a configuration option or does it have to be done 'manually' in program code?

    Shuffle spill behavior is controlled by the property spark.shuffle.spill. It's true (=spill to disk) by default.

Upvotes: 4

Related Questions