CODEWITHSUNDEEP
tensorflowgoogle-cloud-platformgoogle-cloud-ml-engine
reese0106
reese0106

Reputation: 2071

Google Cloud ML Engine GPU Utilization

If I am using --scale-tier BASIC GPU within a Google Cloud ML Engine job, how can I view the GPU utilization? I am able to view CPU Utilization and Memory utilization on the "job details" tab, but I'm wondering how much the GPU is being utilized. Is this just contained within CPU usage or is there another tab to look at GPU utilization?

Additionally, are there any ways to view which ops are taking up most of the CPU usage? My CPU utilization is very high, my memory is very low and my input producer is always full (100%) so I'm trying to get a better understanding of where the time is being spent so that I can try to optimize my model performance.

Upvotes: 5

Views: 1820

Answers (2)

Stephane Bersier
Stephane Bersier

Reputation: 799

watch -n 0.5 nvidia-smi can be used from the command line to see NVIDIA GPU usage.

Upvotes: 2

Jeremy Lewi
Jeremy Lewi

Reputation: 6776

There is currently no way to see GPU utilization with Cloud ML Engine.

TensorFlow has a feature called timelines which can be used to obtain profile data. Here's a blog post describing how to use it.

Upvotes: 4

Related Questions