CODEWITHSUNDEEP
google-cloud-platformgcloud
jason m
jason m

Reputation: 6835

Orchestrating many instances from a local - Too frequent operations from the source resource

I have a local linux machine of the following flavor:

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

On that machine I have gcloud installed with the following version:

$ gcloud -v
Google Cloud SDK 334.0.0
alpha 2021.03.26
beta 2021.03.26
bq 2.0.66
core 2021.03.26
gsutil 4.60

On that machine I have the following shell script:

#!/bin/bash

client_name=$1
instance_name_arg=imageclient${client_name}
z=us-east1-b

gcloud beta compute --project=foo-123 instances create $instance_name_arg --zone=${z} --machine-type=g1-small --subnet=default --network-tier=PREMIUM --source-machine-image projects/foo-123/global/machineImages/envNameimage2

gcloud compute instances start $instance_name_arg --zone=${z}
sleep 10s

gcloud compute ssh --zone=${z} usr@${instance_name_arg} --  '/home/usr/miniconda3/envs/minienvName/bin/python /home/usr/git/bitstamp/envName/client_run.py  --payload=client_packages_'${client_name}

gcloud compute instances stop $instance_name_arg --zone=${z}

As I am scaling this project, I am needing to launch this task many times at exactly the same time but with different settings.

As I am launching more and more of these bash scripts, I am starting to get the following error message:

ERROR: (gcloud.beta.compute.instances.create) Could not fetch resource:
 - Operation rate exceeded for resource 'projects/foo-123/global/machineImages/envNameimage2'. Too frequent operations from the source resource.

How do I work around this issue?

As my solution is architected, I have a "one-machine to one-launch-setting" solution and would ideally want to persist this.

There must be other, large-scale gcloud customers who may or may not need a large number of machines spawned in parallel.

Thank you very much

Upvotes: 6

Views: 1190

Answers (1)

Toni
Toni

Reputation: 1354

As you are using the machine image envNameimage2 for creating the new instance, this is seen as a snapshot of the disk.

You can snapshot your disks at most once every 10 minutes. If you want to issue a burst of requests to snapshot your disks, you can issue at most 6 requests in 60 minutes.

Reference:

A workaround could be to follow the rate limits, or to create an instance using an existing (available) disk with the --disk flag.

--disk=name=clone-disk-1,device-name=clone-disk-1,mode=rw,boot=yes

Upvotes: 4

Related Questions