CODEWITHSUNDEEP
apache-spark
Shengjie
Shengjie

Reputation: 12796

what factors affect how many spark job concurrently

We recently have set up the Spark Job Server to which the spark jobs are submitted.But we found out that our 20 nodes(8 cores/128G Memory per node) spark cluster can only afford 10 spark jobs running concurrently.

Can someone share some detailed info about what factors would actually affect how many spark jobs can be run concurrently? How can we tune the conf so that we can take full advantage of the cluster?

Upvotes: 4

Views: 4266

Answers (2)

Tzach Zohar
Tzach Zohar

Reputation: 37852

Question is missing some context, but first - it seems like Spark Job Server limits the number of concurrent jobs (unlike Spark itself, which puts a limit on number of tasks, not jobs):

From application.conf

 # Number of jobs that can be run simultaneously per context
 # If not set, defaults to number of cores on machine where jobserver is running
 max-jobs-per-context = 8

If that's not the issue (you set the limit higher, or are using more than one context), then the total number of cores in the cluster (8*20 = 160) is the maximum number of concurrent tasks. If each of your jobs creates 16 tasks, Spark would queue the next incoming job waiting for CPUs to be available.

Spark creates a task per partition of the input data, and the number of partitions is decided according to the partitioning of the input on disk, or by calling repartition or coalesce on the RDD/DataFrame to manually change the partitioning. Some other actions that operate on more than one RDD (e.g. union) may also change the number of partitions.

Upvotes: 2

Hamel Kothari
Hamel Kothari

Reputation: 737

Some things that could limit the parallelism that you're seeing:

  • If your job consists of only map operations (or other shuffle-less operations), it will be limited to the number of partitions of data you have. So even if you have 20 executors, if you have 10 partitions of data, it will only spawn 10 task (unless the data is splittable, in something like parquet, LZO indexed text, etc).
  • If you're performing a take() operation (without a shuffle), it performs an exponential take, using only one task and then growing until it collects enough data to satisfy the take operation. (Another question similar to this)

Can you share more about your workflow? That would help us diagnose it.

Upvotes: 0

Related Questions