CODEWITHSUNDEEP
kuberneteskubectl
TheHeroOfTime
TheHeroOfTime

Reputation: 771

Kubernetes: Label each graphic card on a node for being able to schedule a specific pod on that card

I have some nodes with different amount of gpu cards. I want to be able to assign a specific pod to a certain gpu card, something like a reservation when I am assigning a task to it.

I have already seen, that you can label a node like that:

kubectl label node nodaA project-type=ProjectA

But this is just labeling a node, which a pod will be assigned to it. But I want to be able to be more specific.

Let's say I have a node called NodeA, which has 3 GPU cards from Nvidia. The first card is reserved for a specific task called "CertainOne", the other two cards are just there available for other pods.

When a user submits pods which is not the "CertainOne", the pods should be assigned to these 3 cards, no matter what label. But when another user submits a pod with the label "CertainOne", then the reserved card should always be used (the other running pod which could be using at that moment should be dropped and the pod with the label CertainOne should use it as a priority)

But I wasn't able to find a solution for that. The whole idea is, that I can change the reservation card at anytime, for example to another card, but also to change the range, let's say not only the first one, but also the second one, so that pods with the label "CertainOne" should be assigned to card 1 or card 2 in this second scenario.

Is this even possible? Or is there another similar solution to that?

Upvotes: 1

Views: 531

Answers (1)

Vit
Vit

Reputation: 8481

But this is just labeling a node, which a pod will be assigned to it. But I want to be able to be more specific.

But I wasn't able to find a solution for that.

Is this even possible? Or is there another similar solution to that?

Unfortunately, as per my experience, I have bad news for you cause I have never seen solutions like what you want. The very standard way is to use NodeAffinity in spike with appropriate labels and selectors. But this guarantee you scheduling pods on chosen nodes only and this doesnt fit you requirements :(

$ kubectl label nodes nodeA k8s-node=gpunode

apiVersion: v1
kind: Pod
metadata:
  name: runongpunode
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: k8s-node
            operator: In
            values:
            - gpunode

Upvotes: 1

Related Questions