CODEWITHSUNDEEP
google-cloud-platformgoogle-compute-enginegoogle-ai-platform
Hipety Hopit
Hipety Hopit

Reputation: 11

Google Cloud AI Platform training job error: request for 1 K80 accelerators exceeds the allowed maximum of 0. What quota do I need to increase?

I am submitting a Google Cloud AI Platform training job. The job ran fine while I was under my trial period, but now gives the following error:

ERROR: (gcloud.ai-platform.jobs.submit.training) HttpError accessing <https://ml.googleapis.com/v1/projects/[project ID]/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'content-encoding': 'gzip', 'date': 'Mon, 04 Sep 2023 18:38:52 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'transfer-encoding': 'chunked', 'status': 429}>, content <{
  "error": {
    "code": 429,
    "message": "Quota failure for project [project ID]. The request for 1 K80 accelerators exceeds the allowed maximum of 0 A100, 0 K80, 0 P100, 0 P4, 0 T4, 0 TPU_V2, 0 TPU_V2_POD, 0 TPU_V3, 0 TPU_V3_POD, 0 V100 accelerators. To read more about Cloud ML Engine quota, see https://cloud.google.com/ml-engine/quotas.", 
    "status": "RESOURCE_EXHAUSTED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.QuotaFailure",
        "violations": [
          {
            "subject": "[project ID]",
            "description": "The request for 1 K80 accelerators exceeds the allowed maximum of 0 A100, 0 K80, 0 P100, 0 P4, 0 T4, 0 TPU_V2, 0 TPU_V2_POD, 0 TPU_V3, 0 TPU_V3_POD, 0 V100 accelerators."
          }
        ]
      }
    ]
  }
}
>
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.

I am using the us-central-1 region.

Looking at the project quotas on the IAM & Admin dashboard, neither the region K80 quotas, nor the global GPU quota is set to 0. The only quota I can find that's set to 0 is Committed NVIDIA K80 GPUs (default), which does not allow editing.

I previously increased the Committed NVIDIA K80 GPUs quota and the GPUs (all regions) quoate, which were both 0 by default, but this did not change the error.

Are there any other quotas that I need to set or how do I change the Committed NVIDIA K80 GPUs (default) quota?

Upvotes: 1

Views: 74

Answers (0)

Related Questions