CODEWITHSUNDEEP
google-cloud-platformgoogle-kubernetes-engine
Jon
Jon

Reputation: 57

GCP GKE node pool auto schedule

i'm pretty new in GCP and wonder how can I schedule my GKE cluster node pool that contain 3 GPU machines. I found that the operating cost of these machines is very high so I was looking for a solution that could automatically turn off all these machines/pool every night and turn them on every morning as this is a development project.

Upvotes: 0

Views: 894

Answers (1)

dany L
dany L

Reputation: 2654

A possible solution would be to create two nodes pools:

-<default-node-pool> for kube-system pods
-<gpu-node-pool>

The reason why having only one node pool will not work is because a node pool will not scale down to 0, somehow kube-sytem pods need to live somewhere.

You may use the concept of node taint in order to create your gpu node pool with this flag

--node-taints cost=high:NoSchedule

Then you can apply toleration to pods as below

tolerations:
- key: cost
operator: Equal
value: high
effect: NoSchedule

In that way you are making sure that your kube-system pods are not being scheduled on the GPU nodes. There will be some kube-system pods that will run on the nodepool regardless because they have a toleration for everything.

With autoscaler on, and if a node is not needed, gpu node pool should scale down to 0 while keeping default node pool alive for kube-system pods.

Upvotes: 1

Related Questions