CODEWITHSUNDEEP
kubernetesgrafanagrafana-lokitaint
Container-Man
Container-Man

Reputation: 548

Grafana & Loki agents not deployed in Tainted nodes

We are running our workloads on AKS. Basically we have Two Node-Pools.

1. System-Node-Pool: Where all system pods are running

2. Apps-Node-Pool: Where our actual workloads/ apps run in.

In fact, our Apps-Node-Pool is Tainted whereas System-Node-Pool isn't. So basically I deployed Loki-Grafana stack in order for Monitoring and for Log analysis. I'm using below Helm command to install the Grafana-Loki stack.

helm upgrade --install loki grafana/loki-stack  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=standard,loki.persistence.size=5Gi

Since the Toleration isn't added in the helm command (Or even in values.yaml) all the Grafana and Loki pods get deployed in the System-Node-Pool. But my case is, since the necessary agents aren't deployed on Apps-Node-Pool (For example: Promtail Pods) I can't check the logs of my App pods.

Since the Taint exists in Apps-Node-Pool if we add a Toleration along with the Helm command, then basically all the Monitoring related Pods will get deployed in Apps-Node-Pool (Still can't guarant as it may get deployed in System-Node-Pool since it doesn't have a Taint)

So according to my cluster, what can I do in order to make sure that Agent pods are also running in Tainted node?

Upvotes: 0

Views: 1444

Answers (1)

Container-Man
Container-Man

Reputation: 548

So in my case my requirement was to run "Promtail" pods in Apps-Node-Pool. I haven't added a Toleration to the promtail pods, so I had to add a toleration to the Promtail pod. So successfully the Promtail pods got deployed in Apps-Node-Pool.

But still adding a Toleration in Promtail pod doesn't guarant the deployment of the Promtail pods gets deployed in Apps-Node-Pool because in my case, System-Node-Pool didn't have any Taint.

In this case you may leverage both NodeAffinity and Tolerations to exclusively deploy pods in a specific node.

Upvotes: 2

Related Questions