CODEWITHSUNDEEP
kubernetes
pioupiou
pioupiou

Reputation: 946

Pod don't run, insufficient resources

I get this error when i tried to deploy one deployment with ten replicas.

0/2 nodes are available: 1 Insufficient memory, 1 node(s) had taints that the pod didn't tolerate.

I don't understand why two node. Is the same node and just the same problem.

I have a lot of RAM (1GB) free.

How can i fix this error with out add another node.

I have in deployment yaml file this for resources:

limits: cpu: 1000m memory: 1000Mi requests: cpu: 100m memory: 200Mi

Server:

  1. Master: 

    CPU: 2
RAM: 2 - 1 Free

  2. Slave:

    CPU: 2
RAM: 2 - 1 Free

Upvotes: 21

Views: 102609

Answers (3)

Sebinn Sebastian
Sebinn Sebastian

Reputation: 184

Removing the resource request and limit from the POD deployment file worked for me.

Upvotes: 1

raju grj
raju grj

Reputation: 21

I was not able to fix the issue with above answer however, I found a solution.

I want to leave a response so that later if anyone(yes, I mean you) has this same issue and finds this thread won't have to wonder anymore. Basically, your liveness and readiness probe might be making those probes before you pod is actually up and running and keeps restarting in hope that it will be up and running. This will create a cycle and every time, your container restarts same thing happens. Hence, to avoid hitting yourself in the foot, by making a loopback, try to create a delay that will let your container up and running and when liveness and readiness come and make that probe, it will be able to have up status.

    readinessProbe:
      httpGet:
        port: 8000
        path: /actuator/health/readiness
      initialDelaySeconds: 300
      periodSeconds: 30
    livenessProbe:
      httpGet:
        port: 8000
        path: /actuator/health/liveness
      initialDelaySeconds: 300
      periodSeconds: 10

Upvotes: 1

Ohmen
Ohmen

Reputation: 6604

I think you have multiple issues here.

First to the format of the error message you get

0/2 nodes are available: 1 Insufficient memory, 1 node(s) had taints that the pod didn't tolerate.

The first thing is clear you have 2 nodes in total an could not schedule to any of them. Then comes a list of conditions which prevent the scheduling on that node. One node can be affected by multiple issues. For example, low memory and insufficient CPU. So, the numbers can add up to more than what you have on total nodes.

The second issue is that the requests you write into your YAML file apply per replica. If you instantiate the same pod with 100M Memory 5 times they need 500M in total. You want to run 10 pods which request each 200Mi memory. So, you need 2000Mi free memory.

Your error message already implies that there is not enough memory on one node. I would recommend you inspect both nodes via kubectl describe node <node-name> to find out how much free memory Kubernetes "sees" there. Kubernetes always blocks the full amount of memory a pod requests regardless how much this pod uses.

The taints in your error message tells that the other node, possibly the master, has a taint which is not tolerated by the deployment. For more about taints and tolerations see the documentation. In short find out which taint on the node prevents the scheduling and remove it via kubectl taint nodes <node-name> <taint-name>-.

Upvotes: 37

Related Questions