CODEWITHSUNDEEP
google-compute-engine
VS_FF
VS_FF

Reputation: 2363

Script to create multiple GCE VMs simultaneously

I have a basic SH script that I use to create multiple VMs on GCP, and it works fine, but sequentially. When a number of VMs is say above 4 or 5, it becomes a material delay of time. I noticed that in platforms like Dataflow or Dataproc, an arbitrary number of VMs gets created virtually simultaneously. Is there a way to mimic that functionality in GCE? (after all, these seem to be basic GCE machines anyway).

Right now I use the following (simplified) script:

vms=4
for i in `seq 1 $vms`
do

    gcloud compute --project PROJECT disks create VM"$i" --size 50 --zone ZONE --type "pd-ssd"
    gcloud beta compute --project=PROJECT instances create VM"$i" --zone=ZONE --machine-type=MACHINE_TYPE --subnet=default --maintenance-policy=MIGRATE --scopes=https://www.googleapis.com/auth/cloud-platform --disk=name=VM"$i",device-name=VM"$i",mode=rw,boot=yes,auto-delete=yes

done

Thanks for any suggestions!

Upvotes: 2

Views: 2852

Answers (3)

Sirui Sun
Sirui Sun

Reputation: 626

The bulk instance creation APIs allow you to create many VMs with a single API request:

gcloud compute instances bulk create \
  --name-pattern="VM#" \
  --count=4 \
  --region=REGION \
  --machine-type=MACHINE_TYPE \
  --image-project=IMAGE_PROJECT \  # project where your boot disk image is stored
  --image=IMAGE \  # boot disk image name
  --boot-disk-type=pd-ssd \
  --boot-disk-size=50GB \
  --boot-disk-auto-delete \
  --boot-disk-device-name=DEVICE_NAME \  # boot disk device name, the same for all VMs
  --subnet=default \ 
  --maintenance-policy=MIGRATE \
  [...]

Note that:

  • All VMs are created in parallel
  • It automatically chooses the zone in which to create VMs, based on where there's availability
  • It will fail upfront if it detects some issue which prevents creation of the full request (e.g., if there's not enough capacity)
  • It can automatically generate names for you

Upvotes: 0

Mykola Komarevskyy
Mykola Komarevskyy

Reputation: 64

You can create multiple similar VMs faster by creating a group of managed VMs.

First, create an instance template, specifying the VM configuration that you need:

gcloud compute instance-templates create TEMPLATE_NAME \
  --machine-type MACHINE_TYPE \
  --image-project IMAGE_PROJECT \  # project where your boot disk image is stored
  --image IMAGE \  # boot disk image name
  --boot-disk-type pd-ssd \
  --boot-disk-size 50GB \
  --boot-disk-auto-delete \
  --boot-disk-device-name DEVICE_NAME \  # boot disk device name, the same for all VMs
  --subnet default \ 
  --maintenance-policy MIGRATE \
  [...]

Note:

  • You specify boot disk as part of instance template.
  • No need to specify zone for instance template. You will specify desired zone at instance group creation time.
  • Device name is the same for boot disks of all VMs in the group. This is not a conflict because device name of a particular disk is seen from guest OS of each specific VM and is local to that VM.
  • Other parameters are the same as those for creating a VM.

Then, create a group of 4 (or 100, or 1000+) VMs, based on this template:

gcloud compute instance-groups managed create GROUP_NAME \
  --zone ZONE \
  --template TEMPLATE_NAME \ # name of the instance template that you have just created
  --size 4 \ number of VMs that you need to create

The group creates multiple similar VMs, based on your template, much faster than you would do it by iterating creation of standalone VMs.

Upvotes: 4

Yann Coleu
Yann Coleu

Reputation: 1426

Direct way

A quick win would be to add the --async parameters to the gcloud command. Moreover, you can add parallelization in bash with wait and &:

for i in `seq 1 4`
do
  gcloud compute instances [...] --async &
done
wait

Alternative

You can use terraform to do it in a different way

Upvotes: 1

Related Questions