CODEWITHSUNDEEP
kubernetesamazon-eksargocdkubernetes-cronjobhpa
mek
mek

Reputation: 37

Automatic restart of a certain namespace in the cluster and changing the number of replicas

I have a cluster on Amazon AWS EKS with two important namespaces: dev and prod. I'm encountering the following issue with the applications in the prod namespace:

Every night at midnight, all pods in the prod namespace undergo a rollout restart, and after this restart, the default replica count for the applications increases from 1 to 2. There used to be a rollout restart cronjob running at this time, but I have removed it, and currently, there are no cronjobs present. Additionally, there is no other configuration that would increase the replica count to 2. I have confirmed that our Helm charts specify the replica count as 1.

We use Argo CD for our deployment processes, and every night Argo CD falls "out of sync" because the replica count changes.

Here's what I've checked so far:

Cronjobs: Verified that there are no active cronjobs.

Helm Charts: The replica count is set to 1 in the deployments.

HPA (Horizontal Pod Autoscaler): Confirmed that HPA is not being used.

Argo CD Logs: Argo CD falls out of sync every night due to the change in replica count.

What could be the possible causes of this issue and how can I resolve it?

Thanks.

Upvotes: 0

Views: 212

Answers (1)

Chopper
Chopper

Reputation: 89

You can check the control plane logs of the EKS cluster to see who is pathing/updating on deployment. You can use query like below

fields @logStream, @timestamp,verb, @message
| filter objectRef.name like /<your deployment name>/
| filter verb like /create|delete|patch|update/
| sort @timestamp asc

https://repost.aws/knowledge-center/eks-get-control-plane-logs

Upvotes: 0

Related Questions