CODEWITHSUNDEEP
kubernetesgpugoogle-kubernetes-enginenvidianvidia-docker
patrick_corrigan
patrick_corrigan

Reputation: 899

GKE GPU Timesharing Driver Capabilities

I'm running nvidia/cuda:11.8.0-base-ubuntu20.04 on Google Kubernetes Engine using GPU Timesharing on T4 gpus

Checking the driver capabilities I get compute and utility. I was hoping to also get graphics and video. Is this a limitation of Timesharing on GKE?

Upvotes: 0

Views: 360

Answers (1)

Dion V
Dion V

Reputation: 824

It should let you use the resources for graphics and video, however time-sharing GPU are ideal for workloads that are not using a high amount of the resources all the time.

Limitations for using Time-Sharing GPU's on GKE's

  • GKE enforces memory (address space) isolation, performance isolation, and fault isolation between containers that share a physical GPU. However, memory limits aren't enforced on time-shared GPUs. To avoid running into out-of-memory (OOM) issues, set GPU memory limits in your applications. To avoid security issues, only deploy workloads that are in the same trust boundary to time-shared GPUs.
  • GKE might reject certain time-shared GPU requests to prevent unexpected behavior during capacity allocation
  • The maximum number of containers that can share a single physical GPU is 48. When planning your time-sharing configuration, consider the resource needs of your workloads and the capacity of the underlying physical GPUs to optimize your performance and responsiveness.

Upvotes: 1

Related Questions