CODEWITHSUNDEEP
logstash
Justian Meyer
Justian Meyer

Reputation: 3703

Inaccurate Logstash Metric{}

I've been trying to monitor the rate of all my incoming logs in logstash, but it looks like the metrics taken are inaccurate or I'm not understanding their use correctly. I get the current 1m rate of the log, but as can be seen by the total count in the output below, I'm processing around 400 logs every 5s (when the metric flushes), which is much more than the 50-60 per 1m displayed (it eventually levels out around 75/min).

Am I doing something wrong or am I just misinterpreting how metrics work? Is there any way to determine how many logs I'm processing a minute? The final goal is to be able to send email alerts when the number of logs reaches a certain threshold.

Some sample output:

Rate: 3.5552232445405796 (total: 21)
Rate: 55.24209377983976 (total: 3271)
Rate: 55.24209377983976 (total: 3604)
Rate: 56.150221820257926 (total: 3952)
Rate: 57.24159782479909 (total: 4297)
Rate: 58.29368557326007 (total: 4652)
Rate: 59.27764414700567 (total: 5002)
Rate: 60.086983034562124 (total: 5346)
Rate: 60.78363740637835 (total: 5736)
Rate: 62.17617287257932 (total: 6085)
Rate: 62.83371378453949 (total: 6425)
Rate: 63.21480498894082 (total: 6777)
Rate: 63.83727481327493 (total: 7126)
Rate: 64.37799246428055 (total: 7465)
Rate: 64.65160107994187 (total: 7815)
Rate: 64.99927986102034 (total: 8168)
Rate: 65.73492882556424 (total: 8523)
Rate: 66.02797173674145 (total: 8870)
Rate: 66.32956646456502 (total: 9202)
Rate: 66.4471358386394 (total: 9528)
Rate: 66.58728711873627 (total: 9902)
Rate: 67.05204597974938 (total: 10237)

In filter{}

metrics {
        meter => "events"
        add_tag => "metric"
}

In output{}

if "metric" in [tags] {
    if [events.rate_1m] >= 1 {
        stdout {
            codec => line {
                format => "Rate: %{events.rate_1m} (total: %{events.count})"
            }
        }
    }
 }

Upvotes: 1

Views: 462

Answers (1)

Alcanzar
Alcanzar

Reputation: 17155

Logstash uses the Metriks library to generate the metrics. According to it's readme, the rate stuff is in "per second". So the rate_1m is the per second rate over the last minute.

Upvotes: 4

Related Questions