CODEWITHSUNDEEP
apache-spark
Glide
Glide

Reputation: 21245

What happens under the hood when you specify more partitions than the number of cores available?

For example, my laptop has 8 cores and if I submit:

sc.parallelize(1 to 100, 30).count

Does it execute 8 tasks first then the other 22 tasks is executed one by one when a core becomes available?

Is there a way to verify what happens under the hood myself via the SparkUI or logs?

Upvotes: 0

Views: 119

Answers (1)

zsxwing
zsxwing

Reputation: 20826

Does it execute 8 tasks first then the other 22 tasks is executed one by one when a core becomes available?

Yes. Spark UI has an event timeline view to observe it [1]. You can also see when a task started in the Spark logs.

[1] https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html

Upvotes: 1

Related Questions