CODEWITHSUNDEEP
google-cloud-platformgoogle-compute-engine
user2101712
user2101712

Reputation: 115

What happens when a local SSD fails in a Google Cloud Platform compute engine instance?

What happens when one of the multiple local SSDs attached to a compute engine instance has a hardware failure? Specifically:

  1. Is the failure automatically detected by Google Cloud Platform?
  2. Is there a notification, such as by email?
  3. How long does it take for the drive to be replaced?
  4. Is the VM stopped, and restarted after the replacement? Or, is it a hot-swap?
  5. Obviously, the data on that SSD is lost, however, what happens to the data on other SSDs attached to the same Virtual Machine?

Edit: I am aware of the "ephemeral" nature of local SSDs, and plan to replicate my data on multiple machines across different zones in my primary region, and at least one replication to a completely different region. The database I am planning to use is "data-center/rack aware". I am particularly looking for documentation/information about how Google Cloud Platform handles hardware failures in local SSDs.

Upvotes: 4

Views: 1543

Answers (3)

Wojtek_B
Wojtek_B

Reputation: 4443

I'd like to clarify #5.

If your VM experiences host error google documentation states:

If the host system experiences a host error, Compute Engine makes a best effort to reconnect to the VM and preserve the local SSD data, but might not succeed. If the attempt is successful, the VM restarts automatically. However, if the attempt to reconnect fails, the VM restarts without the data.

Which means that you aren't guaranteed to get your data back. Which isn't fun plan accordingly and store your data in more reliable solutions such as persistent disks or buckets.

Upvotes: 1

mensi
mensi

Reputation: 9826

  1. Yes
  2. It depends - block-level failures are just that and are directly passed-through to the guest. So you might see read errors in your dmesg or similar. If an entire device fails, you get a hostError in your Cloud Logging logs for the instance. What happens next depends on your maintenance policy.
  3. Drives are not replaced from the user's point of view - you can only get a new instance. (Of course Google internally replaces broken hardware but this is not exposed to the customer)

Points 4. and 5. are a bit tricky to answer - when an automatic restart for a hostError happens, you have a 60 minutes recovery timeout. This can however mean in practice that your instance is spending 60 minutes in a RUNNING but not booted state while trying to get a broken Local SSD back to then eventually fail and boot up with blank Local SSDs.

Overall, I would recommend you treat an instance as the failure domain and not individual disks as any sort of issue is likely to lead to a hostError of the instance instead of partial failure.

Upvotes: 2

Martin Zeitler
Martin Zeitler

Reputation: 76639

You might want to use persistent disks instead, because your use-case might not apply:

As adding local SSDs reads:

Local SSDs are suitable only for temporary storage such as caches, processing space, or low value data. If you store important data in a local SSD device, you must also store that same data in a durable storage option.

Upvotes: 3

Related Questions