CODEWITHSUNDEEP
prometheusprometheus-alertmanager
ARK
ARK

Reputation: 77

Time based alerts in prometheus alertmanager

I want to set an alert in Prometheus alertmanager that has to be triggered between a given time Eg: Condition to be checked between 09:15 to 15:30

Upvotes: 6

Views: 8319

Answers (2)

Austin Poole
Austin Poole

Reputation: 785

Another option you can you is active_time_intervals in your alertmanager configuration

# Times when the route should be active. These must match the name of a
# time interval defined in the time_intervals section. An empty value
# means that the route is always active.
# Additionally, the root node cannot have any active times.
# The route will send notifications only when active, but otherwise
# acts normally (including ending the route-matching process
# if the `continue` option is not set).
active_time_intervals:
  [ - <string> ...]

so an example of your configuration would be

global:
  ...
route:
  group_by: ['...']
  receiver: 'black-hole-receiver'
  routes:
    - matchers:
        - env=dev
      continue: true
      receiver: 'black-hole-receiver'
      active_time_intervals:
        - workHours

time_intervals:
  - name: workHours
    time_intervals:
      - times:
          - start_time: 09:00
            end_time: 17:00
        weekdays: ['monday:friday']

receivers:
  - name: 'black-hole-receiver'

Upvotes: 2

aclowkay
aclowkay

Reputation: 3877

Alerts in Prometheus are evaluated periodically, and you can't really set a schedule for them.

I think it can be acheived with some PromQL Kong Fu:

scalar(clamp(hour() > 9 and hour() < 15, 1, 1)) * <alert_promql>

hour() > 9 and hour() < 15 Define a range of time based on hour of the day (you can add minutes too)

clamp(..., 1, 1) ensure that the value will be 1 and nothing else

* - This is where the magic happens. If we get any value from the previous function it will be 1 so multiplying by 1 has no effect on the second expression. Otherwise, there is no series on the first expression, so the multiplication will return no results anyway.

Upvotes: 5

Related Questions