Unable to use Compute engine instance

Question

We have a compute engine instance (n1-standard-2, 2 vCPUs, 7.5 GB memory, debian-9-stretch-v20190813 - 30GB) in our project, which we don't run always. Using a cron we start it once a day, instance does it's work, and it shuts down (we use commands to shutdown once job is over). This works perfect for like couple of months, then we notice, instances starts but it does nothing. Doesn't start scripts we intend to start in bootup-script, and further doesn't shut down (if we notice, we need to manually shut it down). Also, we can't connect to it using browser window. Here's the snippet of error in serial port1.

localhost google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'].

Trying to create a new instance by using snapshot of this instance (with increased memory) also, produces same error. We have no choice but to create a fresh new instance, and copy scripts there. This has happened twice. Can someone from compute engine team help? This effects our cost, as we don't monitor instance everyday, and instance keeps running.

Answer 1

This error can appear for to two reasons:

• the boot disk is full or
• the boot disk is set to read-only mode.

As a result, startup scripts can't create temporary files, therefore they don't work; SSH keys can't be stored on the disk, so this causes login failure.

You can try the following options to resolve the issue:

1. Restart the instance. There is a chance that some temporary files will be deleted, so the minimum free space required to run the instance will appear.

2. Attempt to access the instance through the serial console and clean up the /tmp, /var/tmp, /usr/tmp directory manually.

Compute Engine > Doc > Interacting with the serial console

https://stackoverflow.com/a/61485465/11602913

3. Resize your disk manually through serial console: persistent disk, partition and file system.

https://stackoverflow.com/a/61485465/11602913

Compute Engine > Doc > Adding or resizing zonal persistent disks > Resizing a zonal persistent disk

Most Linux instances created using Google public images can automatically resize the root filesystem after the instance's persistent disk has been resized and the instance has been restarted.

4. Create a new instance and select the new boot disk created from the snapshot as a boot disk for this new instance or mount it to a new instance as a secondary disk.

Compute Engine > Doc > Adding or resizing zonal persistent disks > Recovering an inaccessible instance or a full boot disk

Solution 4. in more details.

Stop the instance:

$ gcloud compute instances stop my-vm

Create a snapshot to backup existing state:

Compute Engine > Doc > Creating persistent disk snapshots > Creating a snapshot > Create a snapshot of a regional persistent disk

$ gcloud compute disks snapshot my-vm-disk --snapshot-names my-vm-disk-snap-yyyymmdd --zone=my-zone

Create a new boot disk of a larger size (at least double size is recommended) from the snapshot:

$ gcloud compute disks create my-vm-disk-new --source-snapshot="my-vm-disk-snap-yyyymmdd" --zone=my-zone --size=20 

--size=SIZE Size of the disks. The value must be a whole number followed by a size unit of GB for gigabyte, or TB for terabyte. If no size unit is specified, GB is assumed. If disk size is not specified, the default size of 500GB for standard disks and 100GB for pd-ssd disks will be used.

Create a new instance and select the new boot disk created from the snapshot as a boot disk for the new instance:

$ gcloud compute instances create my-vm-new --zone=my-zone --disk=name=my-vm-disk-new,boot=yes

OR Create a new instance

$ gcloud compute instances create my-vm-new --zone=my-zone

and then attach the new boot disk to the new instance

$ gcloud compute instances attach-disk my-vm-new --disk="my-vm-disk-new"

Restart the new instance:

$ gcloud compute instances reset my-vm-new --zone=my-zone

Connect to the new instance:

$ gcloud compute ssh my-vm-new

Next steps

• Follow recommended practices and consider placing some of the following directories on separate file systems: /home, /opt, /usr, /tmp, /var, /srv.
• Keep an eye on the free space of the file systems. Google Monitoring or any third-party solution could be used.