CODEWITHSUNDEEP
apache-sparkgoogle-cloud-platformserverlessgoogle-cloud-dataprocgoogle-cloud-dataproc-serverless
doeJohn
doeJohn

Reputation: 11

Reducing Dataproc Serverless CPU quota

Aim: I want to run spark jobs on Dataproc Serverless for Spark.

Problem: The minimum CPU cores requirement is 12 cores for a Spark app. That doesn't fit into the default regional CPU quota we have and requires us to expand it. 12 cores is an overkill for us; we don't want to expand the quota.

Details: This link mentions the minimum requirements for Dataproc Serverless for Spark: https://cloud.google.com/dataprocserverless/docs/concepts/properties

They are as follows: (a) 1 driver and 2 executor nodes (b) 4 cores per node

Hence, a total 12 CPU cores is required.

Can we bypass this and run Dataproc Serverless for Spark with less CPU cores?

Upvotes: 1

Views: 336

Answers (1)

Igor Dvorzhak
Igor Dvorzhak

Reputation: 4457

Right now Dataproc Serverless for Spark workload requires 12 CPU cores to run - this is a hard minimum that you can not bypass.

We are working on relaxing this requirement, but it will not be available until at least Q3 2023.

Upvotes: 0

Related Questions