How can I get nested virtualization working on e2-micro GCP VM?

Question

I'm trying to utilize the nested virtualization feature on Google Cloud Platform (GCP), however I was unsuccessful when attempting to use it. Here's the procedure I followed, according to the GCP documentation:

1. Create a new disk, based on Ubuntu 18 LTS image
2. Create a new image from the disk, applying the vmx license
3. Create a new VM instance from the image
4. Check if nested virtualization is supported, using the cpu-checker package in Ubuntu

gcloud compute disks create disk1 --image-project ubuntu-os-cloud --image-family ubuntu-1804-lts --zone us-central1-a
gcloud compute images create nested-vm-image --source-disk disk1 --source-disk-zone us-central1-a --licenses "https://compute.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"
gcloud compute instances create firecracker --description firecracker --image nested-vm-image --zone us-central1-a --preemptible --machine-type e2-micro

Upon running the sudo kvm-ok command, I received this message:

INFO: Your CPU does not support KVM extensions

KVM acceleration can NOT be used

Troubleshooting

Ensure I am Using Haswell CPU

I used the gcloud compute instances describe command to confirm that my e2-micro VM was utilizing the Intel Haswell CPU, which is the minimum required for nested virtualization to work, according to the GCP documentation.

(base) ➜  ~ gcloud compute instances describe firecracker
No zone specified. Using zone [us-central1-a] for instance: [firecracker].
canIpForward: false
cpuPlatform: Intel Haswell
creationTimestamp: 'xxxxxxxxxxxxxxx'
deletionProtection: false
description: firecracker
disks:
- autoDelete: true
  boot: true
  deviceName: persistent-disk-0
  guestOsFeatures:
  - type: VIRTIO_SCSI_MULTIQUEUE
  index: 0
  interface: SCSI
  kind: compute#attachedDisk
  licenses:
  - https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/licenses/ubuntu-1804-lts
  - https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx
  mode: READ_WRITE
  source: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/zones/us-central1-a/disks/firecracker
  type: PERSISTENT
id: 'xxxxxxxxxxxxxxxxxxxxx'
kind: compute#instance
labelFingerprint: xxxxxxxxxxxxxx
machineType: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/zones/us-central1-a/machineTypes/e2-micro
metadata:
  fingerprint: xxxxxxxxxxxxxxx
  kind: compute#metadata
name: firecracker
networkInterfaces:
- accessConfigs:
  - kind: compute#accessConfig
    name: external-nat
    natIP: xxxxxxxxxxx
    networkTier: PREMIUM
    type: ONE_TO_ONE_NAT
  fingerprint: xxxxxxxxxxxxxx
  kind: compute#networkInterface
  name: nic0
  network: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/global/networks/default
  networkIP: xxxxxxxxxxxxxx
  subnetwork: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/regions/us-central1/subnetworks/default
scheduling:
  automaticRestart: false
  onHostMaintenance: TERMINATE
  preemptible: true
selfLink: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/zones/us-central1-a/instances/firecracker
serviceAccounts:
- email: xxxxxxxx@developer.gserviceaccount.com
  scopes:
  - https://www.googleapis.com/auth/devstorage.read_only
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/monitoring.write
  - https://www.googleapis.com/auth/pubsub
  - https://www.googleapis.com/auth/service.management.readonly
  - https://www.googleapis.com/auth/servicecontrol
  - https://www.googleapis.com/auth/trace.append
startRestricted: false
status: RUNNING
tags:
  fingerprint: xxxxxxxxxxxxxxxx
zone: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/zones/us-central1-a

Tried Ubuntu 16.04 LTS

I also tried the same process with Ubuntu 16.04 LTS, according to the documentation, and received the same result as with Ubuntu 18.04 LTS.

Question

Question: How can I successfully spin up a GCP VM that supports nested virtualization? As far as I can tell, I followed the documentation's requirements, but couldn't get it to work.

I realize that the documentation doesn't explicitly indicate that they test nested virtualization with Ubuntu 18 LTS, but that doesn't necessarily mean it won't work.

Answer 1

I've tried the same commands as you did but with Ubuntu 16.04 and 18.04 LTS and my VM also doesn't support KVM at first.

I decided to check documentation and try again:

1. boot disk (the same as yours)

   gcloud compute disks create disk1 --image-project ubuntu-os-cloud \ --image-family ubuntu-1804-lts --zone us-central1-a

2. custom image (the same as yours)

   gcloud compute images create nested-vm-image \ --source-disk disk1 --source-disk-zone us-central1-a \ --licenses "https://compute.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"

3. VM n1-standard-1 (accordingly to the official documentation)

   gcloud compute instances create example-nested-vm --zone us-central1-b \ --min-cpu-platform "Intel Haswell" \ --image nested-vm-image

and nested virtualization works properly:

$ grep -cw vmx /proc/cpuinfo 1 $ sudo kvm-ok INFO: /dev/kvm exists KVM acceleration can be used

Also it works for preemptible VMs:

gcloud compute instances create preemptible-nested-vm \ --zone us-central1-a --preemptible \ --min-cpu-platform "Intel Haswell" \ --image nested-vm-image

I've tried to set another machine-type at step 3, but got an error:

gcloud compute instances create e2-nested-vm --zone us-central1-a \ --machine-type e2-standard-2 \ --min-cpu-platform "Intel Haswell" \ --image nested-vm-image

ERROR: (gcloud.compute.instances.create) Could not fetch resource: - Setting minimum CPU platform is not supported for the selected machine type e2- standard-2.

but this command doesn't follow the documentation.

So, nested virtualization works properly for Ubuntu 16.04 and 18.04 if you follow the guide step by step and machine-type e2-standard-2 doesn't support nested virtualization.

Answer 2

This is a guess answer ... if wrong, post a comment and it will be deleted ...

If we look here under Machine Types we find the following:

Shared-core machine types are available in the N1 and E2 families. These machine types timeshare a physical core. This can be a cost-effective method for running small, non-resource intensive applications.

• N1: f1-micro and g1-small shared-core machine types have up to 1 vCPU available for short periods of bursting.

• E2: e2-micro, e2-small, and e2-medium shared-core machine types have 2 vCPUs available for short periods of bursting.

In your story you were saying you were using e2-micro machine type. Machine this all together we see that you have chosen a machine type that appears to perform shared-core processing and are trying to enable nested virtualization. My gut is saying that shared core means that your CPU running your OS/app is time sliced between other users running their OS/apps ... while the concept of nested virtualization (I think) requires that you have exclusive ownership of your CPU as is found in "normal" family members. I'd suggest running your recipe using an n1-standard-2 machine type and see if the same recipe that you applied works. If it does ... then we might be able to draw a tentative conclusion that micro/small with shared core processors might not support nested virtualization.