CODEWITHSUNDEEP
tensorflowgoogle-cloud-platformdeep-learninggpunvidia
djvaroli
djvaroli

Reputation: 1423

Google Cloud Deep Learning On Linux VM throws Unknown Cuda Error

I am trying to set up a deep learning VM on Google Cloud but I keep running into the same issue over and over again.

I will follow all the steps, set up a N1-highmem-8 (8 vCPU, 52gb Memory) instance, add a single T4 GPU and select the Deep Learning Image: TensorFlow 2.4 m69 CUDA 110 image. That's it.

After that, I will ssh into the vm, run the script that installs all the NVIDIA drivers and... when I begin using it, by simply running

from tensorflow.keras.layers import Input, Dense

i = Input((100,))
x = Dense(500)(i)

I keep getting failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error. By that point I haven't installed anything and haven't done anything custom, just the vanilla image from GCP.

What is more concerning is that, even if I delete the vm and then create a new one with the same config, some times the error won't happen immediately and sometimes it's present off the bat.

Has anyone encountered this? I've googled around to see if anyone has faced this issue and while I came across suggestions, all of them are old and have not worked for me. More over, suggestions on NVIDIA support forums tell me to re-install everything and the whole point of me using a pre-built GCP image specifically for deep learning is so that I don't have to enter the hell of installing and resolving issues with NVIDIA drivers.

Upvotes: 1

Views: 403

Answers (1)

Chandra Kiran Pasumarti
Chandra Kiran Pasumarti

Reputation: 702

The issue is fixed with the M74 image, but you are using M69. So follow one of the two fixes provided in the Google Cloud public forum.

we can mitigate the issue by:

Fix #1: Use the latest DLVM image (M74 or later) in a new VM instance: They have released a fix for the newest DLVM image in M74 so you will no longer be affected by this issue.

Fix #2: Patch your existing instance running images older than M74.

Run the following via an SSH session on the affected instance:
gsutil cp gs://dl-platform-public-nvidia/b191551132/restart_patch.sh /tmp/restart_patch.sh
chmod +x /tmp/restart_patch.sh
sudo /tmp/restart_patch.sh
sudo service jupyter restart

This only needs to be done once, and does not need to be rerun each time the instance is rebooted.

Upvotes: 2

Related Questions