CODEWITHSUNDEEP
kubernetes
karunasagar
karunasagar

Reputation: 31

How to prevent repeated reconcile calls

I'm developing a Kubernetes controller. The desired state for this controller is captured in CRD-A and then it creates a deployment and statefulset to achieve the actual state. Currently I'm using server side apply to create/update these deployment and statefulsets.

The controller establishes watch on both CRD-A as well as deployments, statefulset. This to ensure that if there is a change in the deployment/statefulset, the reconcile() is notified and takes action to fix it. Currently the reconcile() always calls server side apply to create/update and this leads another watch event (resource version changes on every server side apply) resulting in repeated/infinite calls to reconcile()

One approach I've been thinking about is to leverage 'generation' on deployment/statefulset i.e. the controller will maintain a in-memory map of (k8s object -> generation) and on reconcile() compare the value in this map to what is present in the indexed informer cache; do you see any concerns with this approach? And are there better alternatives to prevent repeated/infinite reconcile() calls?

Upvotes: 3

Views: 2996

Answers (2)

Keven Deng
Keven Deng

Reputation: 31

Ideally, if the object you provide in the server side apply is not changed, the generation and the resourceVersion of the object should BOTH NOT be changed.

But sometimes that's not the case, see this github issue:https://github.com/kubernetes/kubernetes/issues/95460

Fortunately, the generation always stays the same, so yes, you can leverage this field to avoid the reconcile dead loop, by adding a GenerationChangedPredicate filter to your controller, which will skip reconciling if the generation does not change, and it's often used in conjuction with the LabelChangedPredicate, which filters events when the object's labels does not change.

Here's how you will set up your controller with these two predicates:

ctrl.NewControllerManagedBy(mgr).
    For(&Object{}).
    Owns(&appsv1.StatefulSet{}).
    Owns(&appsv1.Deployment{}).
    // together with Server Side Apply
    // this predicates prevents meaningless reconcilations from being triggered
    WithEventFilter(predicate.Or(predicate.GenerationChangedPredicate{}, predicate.LabelChangedPredicate{})).
    Complete(r)

Upvotes: 3

whites11
whites11

Reputation: 13260

One idea could be to not apply the objects if there are no changes to be applied. This gives you the problem of how to detect if the objects are up to date already, but it's hard to tell if that's doable in your case.

Upvotes: 1

Related Questions