CODEWITHSUNDEEP
kubernetesmonitoringprometheushigh-availability
Luís Marques
Luís Marques

Reputation: 21

Prometheus HA in Kubernetes (AKS)

I'm running the following helm chart (https://github.com/helm/charts/tree/master/stable/prometheus) with server.replicaCount =2 and server.statefulSet.enabled = true.

For storage i use two Manage Disks (not Azure Files that is not POSIX) (2 PV and 2 PVC) are created during the deployment of the chart.

My question is: Is this an HA solution? Are the metrics written to both prometheus instances (a service with a public ip and and headless "service" are created) and replicated to both disks?

How this replicas really work?

Thanks,

Upvotes: 2

Views: 2392

Answers (2)

triwats
triwats

Reputation: 306

Sadly, as Piotr noted, this is not a true HA offering and Thanos is generally the preferred way to go for this kind of setup, but not without it's own gotchas. The amount of clusters you have is a factor, and you might need some sort of tooling account to be able to follow changes all the way through.

What I can offer you is this excellent talk, which includes a live demo and shows how this works in practice.

Upvotes: 1

Piotr Malec
Piotr Malec

Reputation: 3647

No, this is not HA solution. This only scales the deployment to have 2 replicas at all times which both are on statefulsets.

In order to achieve HA monitoring on Kuberetes there needs to be dynamic failure detection and routing tools involved.

There are couple of articles about getting prometheus work with HA:

Deploying an HA Prometheus in Kubernetes on AWS — Multiple Availability Zone Gotchas

HA Kubernetes Monitoring using Prometheus and Thanos

The number of replicas only instructs deployment to always have at least 2 running instances of the deployment pods. You can find more information about replicas in Kubernetes documentation.

In the helm chart documentation, there seems to be other options like server.service.statefulsetReplica.enabled and server.service.statefulsetReplica.replica but I think those are just tools that can help to create HA prometheus. Not a ready from the get go solution.

Hope it helps.

Upvotes: 0

Related Questions