CODEWITHSUNDEEP
kubernetesamazon-eks
alexarsh
alexarsh

Reputation: 5391

How to fix ephemeral local storage problem?

I'm Running some deployment on EKS k8s 1.16 and after ~5 min my pod gets Evicted with the following message:

Pod ephemeral local storage usage exceeds the total limit of containers 1Gi.

My node has 20Gi ephemeral storage.

My QoS Class is Guaranteed and no matter which amount of ephemeral-storage I configure in my yaml, I see the same error with the amount I configure.

Do you have a clue what can be done?

My yaml file is here: https://slexy.org/view/s2096sex7L

Upvotes: 9

Views: 26169

Answers (2)

user3243989
user3243989

Reputation: 101

If you're reading this and you're using GKE Autopilot, there is a hard limit of 10G for ephemeral storage in Autopilot. I would recommend moving your storage to a Volume.

See Autopilot documentation here

Upvotes: 7

Kamol Hasan
Kamol Hasan

Reputation: 13466

It's because you're putting an upper limit of ephemeral-storage usage by setting resources.limits.ephemeral-storage to 1Gi. Remove the limits.ephemeral-storage if safe, or change the value depending upon your requirement.

      resources:
        limits:
          memory: "61Gi"
          cpu: "7500m"
          ephemeral-storage: "1Gi" <----- here
        requests:
          memory: "61Gi"
          cpu: "7500m"
          ephemeral-storage: "1Gi"

If the node where a Pod is running has enough of a resource available, it’s possible (and allowed) for a container to use more resources than its request for that resource specifies. However, a container is not allowed to use more than its resource limit.

Upvotes: 8

Related Questions