CODEWITHSUNDEEP
kubernetes
Chandu
Chandu

Reputation: 2049

How to free storage on node when status is "Attempting to reclaim ephemeral-storage"?

I have a 3 node Kubernetes cluster used for development. One of the node's status is "Attempting to reclaim ephemeral-storage" since 11 days. How to reclaim storage ? Since it is just development instance I cannot extend the storage. I dont care about the existing data in the storage. How to clear the storage ? Thanks

Upvotes: 1

Views: 5336

Answers (3)

Swavek L
Swavek L

Reputation: 21

I was facing errors on one of the nodes which prevented successful deployment. I was seeing failures due to disk-pressure taints. This apparently meant that I was running of of ephemeral storage disk space. I tried to exec into the failing pod but was not able to and consequently I was not able to see which files were eating up space. This article helped a little to explain what is going on.

https://www.airplane.dev/blog/kubernetes-disk-pressure

However the true solution was to use the drain command. I used

kubectl get nodes

to get nodes ids.

Then I drained the pods on the failing node and then restarted them using 'uncordon' command. I did it a few times on both failing and normal node.

Thanks for the tip.

Upvotes: 0

P Ekambaram
P Ekambaram

Reputation: 17621

Just run 'docker system prune command' to free up the space on the node. refer the below command

$ docker system prune -a --volumes

WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all volumes not used by at least one container
        - all images without at least one container associated to them
        - all build cache
Are you sure you want to continue? [y/N] y

Upvotes: 6

Arghya Sadhu
Arghya Sadhu

Reputation: 44569

Since it's a development environment you can just drain the node to clear all pods and their data and then uncordon for pods to be scheduled again

kubectl drain --delete-local-data --ignore-daemonsets $NODE_NAME && kubectl uncordon $NODE_NAME

--delete-local-data flag is for cleaning data of the pods.

Upvotes: 5

Related Questions