CODEWITHSUNDEEP
multithreadingscalaapache-spark
smz
smz

Reputation: 505

Number of CPUs per Task in Spark

I don't quite understand spark.task.cpus parameter. It seems to me that a “task” corresponds to a “thread” or a "process", if you will, within the executor. Suppose that I set "spark.task.cpus" to 2.

  1. How can a thread utilize two CPUs simultaneously? Couldn't it require locks and cause synchronization problems?

  2. I'm looking at launchTask() function in deploy/executor/Executor.scala, and I don't see any notion of "number of cpus per task" here. So where/how does Spark eventually allocate more than one cpu to a task in the standalone mode?

Upvotes: 31

Views: 18736

Answers (2)

sangoro
sangoro

Reputation: 1

if I have set spark.task.cpus to 4 and spark.executor.cores to 4 -- this would mean I am spinning up one task per executor. Is this understanding correct?

Upvotes: -1

marios
marios

Reputation: 8996

To the best of my knowledge spark.task.cpus controls the parallelism of tasks in you cluster in the case where some particular tasks are known to have their own internal (custom) parallelism.

In more detail: We know that spark.cores.max defines how many threads (aka cores) your application needs. If you leave spark.task.cpus = 1 then you will have #spark.cores.max number of concurrent Spark tasks running at the same time.

You will only want to change spark.task.cpus if you know that your tasks are themselves parallelized (maybe each of your task spawns two threads, interacts with external tools, etc.) By setting spark.task.cpus accordingly, you become a good "citizen". Now if you have spark.cores.max=10 and spark.task.cpus=2 Spark will only create 10/2=5 concurrent tasks. Given that your tasks need (say) 2 threads internally the total number of executing threads will never be more than 10. This means that you never go above your initial contract (defined by spark.cores.max).

Upvotes: 34

Related Questions