CODEWITHSUNDEEP
google-cloud-platformcloudgoogle-compute-enginequota
Coder
Coder

Reputation: 607

GCP: Instance creation failed

I recently tried to create an instance group on the Google Cloud Platform (GCP) with 50 n1-standard-1 instances in zone us-east1-b, each with P100 GPUs. I requested and got approval for 200 P100 GPUs in this zone. My CPU, IP addresses, and Routes for this zone and globally all meet the quotas listed on this page.

However, right now, I'm only up to 21 of these 50 instances created, with the rest with a yellow hazard sign and the accompanying warning message: Instance 'instance-group-1-<name>' creation failed: The zone 'projects/<project>/zones/us-east1-b' does not have enough resources available to fulfill the request. '(resource type:compute)'.

Is there any place on the quotas page where I can get information on exactly which compute quota I forgot to ask more of? The error message is unfortunately not very descriptive.

Note: I suspect that this could be a reference to exceeding the Compute Engine API query limiting access to at most 2000 querues per 100 seconds. The 7 day peak usage column does show that I have exceeded it at peak times. However, my Current Usage is at less than 70 queries per 1000 seconds. When I look at my compute engine query usage graphed over time, it doesn't look like I have tripped the 2000 rate limit for several hours. However, the instance group still fails to populate fully to all 50 instances.

Upvotes: 4

Views: 1548

Answers (1)

Notauser
Notauser

Reputation: 406

This is a typical error which means that at a certain point in time, the resources in 'us-east1-b' are not sufficient to scale your Instance Group even though you have sufficient quota. You have two alternatives:

1- try again later 2- request GPUs in another region/zone and deploy your IG there.

Google also recommends to distribute your workloads in more than one region and zone.

For more information see this and this.

Upvotes: 6

Related Questions