Munin alert/notification's not being executed

Question

I have a custom notification script which I want to supply with data from munin whenever a critical event occurs. Unfortunately I was not able to get it working, while following the official docs. The notification script it self is tested and works fine when called from the shell with fake data. It's permissions are on 755 so execution shouldn't be an issue either. Hence the contact hook probably not being called. What I did is this:

# /etc/munin/munin-conf.d/custom.cnf
    [...]
    contact.slack.command MUNIN_SERVICESTATE="${var:worst}" MUNIN_HOST="${var:host}" MUNIN_SERVICE="${var:graph_title}" MUNIN_GROUP=${var:group} /usr/local/bin/notify_slack_munin
    contact.slack.always_send warning critical
    contact.slack.text ${if:cfields \u000A* CRITICALs:${loop<,>:cfields  ${var:label} is ${var:value} (outside range [${var:crange}])${if:extinfo : ${var:extinfo}}}.}${if:wfields \u000A* WARNINGs:${loop<,>:wfields  ${var:label} is ${var:value} (outside range [${var:wrange}])${if:extinfo : ${var:extinfo}}}.}${if:ufields \u000A* UNKNOWNs:${loop<,>:ufields  ${var:label} is ${var:value}${if:extinfo : ${var:extinfo}}}.}${if:fofields \u000A* OKs:${loop<,>:fofields  ${var:label} is ${var:value}${if:extinfo : ${var:extinfo}}}.}

Above those lines the nodes and dir's are defined which works fine. But the notification isn't going out. Do you have any idea on what it could be?

Answer 1

I just played with the same script found originally on gist and got it working after fixing two trivial mistakes:

• make sure that you put the contacts in the config before the host tree. I put it at the end of the configuration file and it wasn't called at all until I moved it to where the sample contacts are placed
• make sure that the command in the Munin configuration uses the full filename, so if you put the script in /usr/local/bin/notify_slack_munin then check that there is no file extension on the file

As written in the Munin Guide, watch the munin-limits.log to see what is happening.

Final note, if you have contact.slack.always_send warning critical it will push a notification repeatedly, not just on severity changes.

For reference (in case the gist link brakes), the script that calls the Slack webhook is the following (slightly modified for clarification):

#!/bin/bash

# Slack notification script for Munin
# Mark Matienzo (@anarchivist)
# https://gist.github.com/anarchivist/58a905515b2eb2b42fe6
#
# To use:
# 1) Create a new incoming webhook for Slack
# 2) Edit the configuration variables that start with "SLACK_" below
# 3) Add the following to your munin configuration before the host tree
#    in the part where sample contacts are listed:
#
# # -- Slack contact configuration 
# # notify_slack_munin.sh is the full file name
# contact.slack.command MUNIN_SERVICESTATE="${var:worst}" MUNIN_HOST="${var:host}" MUNIN_SERVICE="${var:graph_title}" MUNIN_GROUP=${var:group} /usr/local/bin/notify_slack_munin.sh
# # This line will spam Slack with notifications even if no state change happens
# contact.slack.always_send warning critical
# # note: This has to be on one line for munin to parse properly
# contact.slack.text ${if:cfields \u000A* CRITICALs:${loop<,>:cfields  ${var:label} is ${var:value} (outside range [${var:crange}])${if:extinfo : ${var:extinfo}}}.}${if:wfields \u000A* WARNINGs:${loop<,>:wfields  ${var:label} is ${var:value} (outside range [${var:wrange}])${if:extinfo : ${var:extinfo}}}.}${if:ufields \u000A* UNKNOWNs:${loop<,>:ufields  ${var:label} is ${var:value}${if:extinfo : ${var:extinfo}}}.}${if:fofields \u000A* OKs:${loop<,>:fofields  ${var:label} is ${var:value}${if:extinfo : ${var:extinfo}}}.}


SLACK_CHANNEL="#insert-your-channel"
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/insert/your/hookURL"
SLACK_USERNAME="munin"
SLACK_ICON_EMOJI=":munin:"

# If you want to test the script, you may have to comment this out to avoid hanging console
input=`cat`

#Set the message icon based on service state
if [ "$MUNIN_SERVICESTATE" = "CRITICAL" ]
then
    ICON=":exclamation:"
    COLOR="danger"
elif [ "$MUNIN_SERVICESTATE" = "WARNING" ]
then
    ICON=":warning:"
    COLOR="warning"
elif [ "$MUNIN_SERVICESTATE" = "ok" ]
then
    ICON=":white_check_mark:"
    COLOR="good"
elif [ "$MUNIN_SERVICESTATE" = "OK" ]
then
    ICON=":white_check_mark:"
    COLOR="good"
elif [ "$MUNIN_SERVICESTATE" = "UNKNOWN" ]
then
    ICON=":question:"
    COLOR="#00CCCC"
else
    ICON=":white_medium_square:"
    COLOR="#CCCCCC"
fi

# Generate the JSON payload
PAYLOAD="{\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"icon_emoji\": \"${SLACK_ICON_EMOJI}\", \"attachments\": [{\"color\": \"${COLOR}\", \"fallback\": \"Munin alert - ${MUNIN_SERVICESTATE}: ${MUNIN_SERVICE} on ${MUNIN_HOST}\", \"pretext\": \"${ICON} Munin alert - ${MUNIN_SERVICESTATE}: ${MUNIN_SERVICE} on ${MUNIN_HOST} in ${MUNIN_GROUP} - <http://central/munin/|View Munin>\", \"fields\": [{\"title\": \"Severity\", \"value\": \"${MUNIN_SERVICESTATE}\", \"short\": \"true\"}, {\"title\": \"Service\", \"value\": \"${MUNIN_SERVICE}\", \"short\": \"true\"}, {\"title\": \"Host\", \"value\": \"${MUNIN_HOST}\", \"short\": \"true\"}, {\"title\": \"Current Values\", \"value\": \"${input}\", \"short\": \"false\"}]}]}"

#Send message to Slack
curl -sX POST -o /dev/null --data "payload=${PAYLOAD}" $SLACK_WEBHOOK_URL 2>&1