CODEWITHSUNDEEP
pythondjangogoogle-compute-engineautoscalingprometheus
Andy
Andy

Reputation: 1231

Prometheus to track requests in auto-scaled servers

I am trying to use Prometheus to track the number of requests to my server over time. Since my servers will be auto-scalled horizontally using Google Compute Engine, I can only push my metric to the remote push gateway. My servers will be deleted and re-created at any given time.

The problem is that whenever the new server is created, or even the counter instance is created using the python client library, the count value is reset to 0. I can also see the graph goes up and down, instead of always increasing.

enter image description here

What is the proper way to track the total number of requests using Prometheus when in an auto-scalled environment?

EDIT:

There is another post about the exactly the same problem, just in a little different scenario. Prometheus how to handle counters on server. It seems the servers must somehow track the counter state by themselves. Prometheus only record whatever values sent to it at that point, push or pull. Which means the counter value does not always go up if the servers simply call counter.inc(). In other words, the following statement in the document only apply in the client library side.

A counter is a cumulative metric that represents a single numerical value that only ever goes up.

Upvotes: 3

Views: 646

Answers (1)

brian-brazil
brian-brazil

Reputation: 34112

Since my servers will be auto-scalled horizontally using Google Compute Engine, I can only push my metric to the remote push gateway. My servers will be deleted and re-created at any given time.

That's not quite true. You can use service discovery to automatically discover your nodes and have them instrumented and monitored in the usual Prometheus fashion.

The pushgateway is only intended for service-level batch jobs, see https://prometheus.io/docs/practices/pushing/

Upvotes: 2

Related Questions