How do reclaim policys for persistent storage work in Kubernetes?

Question

I am having a bit of a problem understanding how the reclaim policy of persistent storage works in detail. I want to define a storage class that saves some logs and if the pods fail or I restart the setup, keeps the data.

According to here when I set the reclaim policy to delete the PV and the storage in Azure is removed. If it is set to retain the PV needs to be manually reclaimed.

Now I have the feeling that both types are wrong for my usage (although it seems rather common), as I do want to keep the data (so delete is wrong) and I do not want to manually reclaim the PV every time (so retain seems wrong).

Is there something I misunderstand or is there another way to achieve this?

Answer 1

Is there another way to achieve this?

You haven't fully specified your use case but I would consider moving away from directly managing the lifecycle of the PVCs and focusing on the lifecycle/protection of the StatefulSet itself alongside using the PVC retention policy that was introduced in 1.23.

With this setup you need only to worry about the events of:

(1) Scale down of the StatefulSet.
(2) Deletion of the StatefulSet.

For each one of those two events you can choose if you want to delete or retain the PVC.

From here:

If you enable the alpha feature, a StatefulSet spec includes a PersistentVolumeClaim retention policy. This is used to control if and when PVCs created from a StatefulSet’s volumeClaimTemplate are deleted. This first iteration of the retention policy contains two situations where PVCs may be deleted.

The first situation is when the StatefulSet resource is deleted (which implies that all replicas are also deleted). This is controlled by the whenDeleted policy.

The second situation, controlled by whenScaled is when the StatefulSet is scaled down, which removes some but not all of the replicas in a StatefulSet. In both cases the policy can either be Retain, where the corresponding PVCs are not touched, or Delete, which means that PVCs are deleted. The deletion is done with a normal object deletion, so that, for example, all retention policies for the underlying PV are respected.

So:

This policy forms a matrix with four cases. I’ll walk through and give an example for each one.

(1) whenDeleted and whenScaled are both Retain. This matches the existing behavior for StatefulSets, where no PVCs are deleted. This is also the default retention policy. It’s appropriate to use when data on StatefulSet volumes may be irreplaceable and should only be deleted manually.

(2)whenDeleted is Delete and whenScaled is Retain. In this case, PVCs are deleted only when the entire StatefulSet is deleted. If the StatefulSet is scaled down, PVCs are not touched, meaning they are available to be reattached if a scale-up occurs with any data from the previous replica. This might be used for a temporary StatefulSet, such as in a CI instance or ETL pipeline, where the data on the StatefulSet is needed only during the lifetime of the StatefulSet lifetime, but while the task is running the data is not easily reconstructible. Any retained state is needed for any replicas that scale down and then up.

(3) whenDeleted and whenScaled are both Delete. PVCs are deleted immediately when their replica is no longer needed. Note this does not include when a Pod is deleted and a new version rescheduled, for example when a node is drained and Pods need to migrate elsewhere. The PVC is deleted only when the replica is no longer needed as signified by a scale-down or StatefulSet deletion. This use case is for when data does not need to live beyond the life of its replica. Perhaps the data is easily reconstructable and the cost savings of deleting unused PVCs is more important than quick scale-up, or perhaps that when a new replica is created, any data from a previous replica is not usable and must be reconstructed anyway.

(4) whenDeleted is Retain and whenScaled is Delete. This is similar to the previous case, when there is little benefit to keeping PVCs for fast reuse during scale-up. An example of a situation where you might use this is an Elasticsearch cluster. Typically you would scale that workload up and down to match demand, whilst ensuring a minimum number of replicas (for example: 3). When scaling down, data is migrated away from removed replicas and there is no benefit to retaining those PVCs. However, it can be useful to bring the entire Elasticsearch cluster down temporarily for maintenance. If you need to take the Elasticsearch system offline, you can do this by temporarily deleting the StatefulSet, and then bringing the Elasticsearch cluster back by recreating the StatefulSet. The PVCs holding the Elasticsearch data will still exist and the new replicas will automatically use them.

Answer 2

A reclaim policy of delete won't be deleted when the pod is deleted. This is more about when the PVC itself is deleted what happens with the cloud provider.

Delete means if the PVC itself is deleted the provisioned disk is also removed. If you have a pod which is deleted, as long as future pods use the pvc you'll still have the data, just don't delete the pvc resource also.

Having it set to retain means even when the pv/pvc resource is deleted the provisioned storage will remain just in case