CODEWITHSUNDEEP
dockersmtpalertwebhooksprometheus-alertmanager
tandathuynh148
tandathuynh148

Reputation: 308

Alertmanager container can't send SMTP email because the connection refused from web hook

I am using Prometheus/Grafana/Alertmanager docker images for monitoring my service. I created the rules.yml and added it into the prometheus.yml. It worked fine, I saw the ALERTS metrics when the system was overload. Then these alerts were sent to alermanager, the alertmanager was not able to send the messages to SMTP server.

It showed the error

level=warn ts=2021-03-25T07:07:17.526Z caller=notify.go:674 component=dispatcher receiver=web.hook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post "http://127.0.0.1:5001/": dial tcp 127.0.0.1:5001: connect: connection refused"

I think the error was related to the web_hook configuration but I did not set it in my alertmanager.yml, I was using email_config. I tested the alermanager.yml by creating the system service, it sent emails to the receiver when an alert was triggered so I am quite sure my configuration is not wrong.

Here is my configruration in alertmanager.yml

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'smtp.sendgrid.net:587'
  smtp_from: '[email protected]>'
  smtp_auth_username: 'apikey'
  smtp_auth_password: 'sample'

route:
  group_by: [alertname]
  # default receive
  receiver: default-alert
  group_wait: 30s
  group_interval: 2m
  repeat_interval: 3h
  routes:
    - match:
        alertname: PrometheusTargetMissing
      receiver: demo-alert

receivers:
- name: 'demo-alert'
  email_configs:
  - to: '[email protected]'
    # send_resolved: true
    headers: 
      Subject: "system alert"

Do you met this error before? I really approciate your help.

Upvotes: 1

Views: 3666

Answers (1)

tandathuynh148
tandathuynh148

Reputation: 308

Fixed. This error was caused due to missing t charater in the volume docker-compose file

    volumes:
      - alertmanager:/***alermanager***

Upvotes: 0

Related Questions