CODEWITHSUNDEEP
kubernetesdaskworker
Vinh Tran
Vinh Tran

Reputation: 189

Dask Helm Chart - How to create Dask-CUDA-Worker nodes

I installed the Dask Helm Chart with a revised values.yaml to have 10 workers, however instead of Dask Workers I want to create Dash CUDA Workers to take advantage of the NVIDIA GPUs on my multi-node cluster.

I tried to modify the values.yaml as follows to get Dask CUDA workers instead of Dask Workers, but the worker pods are able to start. I did already install the NVIDIA GPUs on all my nodes on the Kubernetes per the official instructions so I'm not sure what DASK needs to see in order to create the Dask-Cuda-Workers.

worker:
  name: worker
  image:
    repository: "daskdev/dask"
    tag: 2.19.0
    pullPolicy: IfNotPresent
    dask_worker: "dask-cuda-worker"
    #dask_worker: "dask-worker"
    pullSecrets:
    #  - name: regcred
  replicas: 15
  default_resources:  # overwritten by resource limits if they exist
    cpu: 1
    memory: "4GiB"
  env:
  #  - name: EXTRA_APT_PACKAGES
  #    value: build-essential openssl
  #  - name: EXTRA_CONDA_PACKAGES
  #    value: numba xarray -c conda-forge
  #  - name: EXTRA_PIP_PACKAGES
  #    value: s3fs dask-ml --upgrade
  resources: {}
  #  limits:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
  #  requests:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1

Upvotes: 1

Views: 208

Answers (1)

quasiben
quasiben

Reputation: 1464

As dask-cuda-worker is not yet officially in the dask image you will need to pull the image a different image: rapidsai/rapidsai:latest

Upvotes: 1

Related Questions