Custom alerts sent to Alertmanager not showing up in Grafana

Question

I am currently using a version of the kube-prometheus-stack Helm chart to deploy Grafana, Prometheus and Alertmanager.

I made a Go program to send custom alerts to Alertmanager (this feature will be used in the future). It looks like this:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

type Alert struct {
    Status         string                `json:"status"`
    Labels         map[string]string     `json:"labels"`
    Annotations    map[string]string     `json:"annotations"`
    EndsAt         string                `json:"endsAt"`
}

func main() {
    url := "http://alertmanager_url:9093/api/v1/alerts"

    alert := []Alert{
        {
            Status: "firing",
            Labels: map[string]string{
                "alertname": "testAlert",
                "severity": "critical",
            },
            Annotations: map[string]string{
                "summary": "This is a test alert.",
            },
            EndsAt: "2023-03-21T00:10:53-03:00",
        },
    }

    jsonData, err := json.Marshal(alert)
    if err != nil {
        return fmt.Println(err)
    }

    client := &http.Client{
        InsecureSkipVerify: true,
    }

    req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData))
    if err != nil {
        return fmt.Println(err)
    }
    req.Header.Set("Content-Type", "application/json")

    resp, err := client.Do(req)
    if err != nil {
        return fmt.Println(err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return fmt.Println(err)
    }

    fmt.Println(string(body))
}

This works fine, I see my alerts popping up in Alertmanager. However, in Grafana, I can see all the standard alerts firing, but not my test alerts.

Is there something wrong with my Go program, or am I missing something? I can provide more information if needed.

Answer 1

I suspect (!?) that Grafana is pulling alert data from Prometheus not from AlertManager. You would need to have Prometheus originate alerts for them to be caught by Grafana.

Your code doesn't compile and you may want to switch to v2.

See Sending Alerts.

You should use api/v2 (defined here)

Try this:

• I've preserved the bulk of your code
• This variant uses /api/v2
• The message (Alerts) differs slightly
• Go JSON marshaling permits conversion to|from time.Time (see StartsAt and EndsAt)
• Your TLS client was created incorrectly

package main

import (
    "bytes"
    "crypto/tls"
    "encoding/json"
    "flag"
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "time"
)

type Alert struct {
    StartsAt     time.Time         `json:"startsAt"`
    EndsAt       time.Time         `json:"endsAt"`
    Annotations  map[string]string `json:"annotations"`
    Labels       map[string]string `json:"labels"`
    GeneratorURL string            `json:"generatorURL"`
}

var (
    endpoint = flag.String(
        "endpoint",
        "0.0.0.0:9093",
        "Endpoint of the AlertManager",
    )
)

func main() {
    flag.Parse()

    startAt := time.Now()
    endsAt := startAt.Add(24 * time.Hour)

    alert := []Alert{
        {
            StartsAt: startAt,
            EndsAt:   endsAt,
            Labels: map[string]string{
                "alertname": "testAlert",
                "severity":  "critical",
            },
            Annotations: map[string]string{
                "summary": "This is a test alert.",
            },
            GeneratorURL: "http://foo/test",
        },
    }

    jsonData, err := json.Marshal(alert)
    if err != nil {
        slog.Error("Unable to marshal", "err", err)
    }

    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                InsecureSkipVerify: true,
            },
        },
    }

    url := fmt.Sprintf("http://%s/api/v2/alerts", *endpoint)
    req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData))
    if err != nil {
        slog.Error("unable to create request",
            "err", err,
        )
    }
    req.Header.Set("Content-Type", "application/json")

    resp, err := client.Do(req)
    if err != nil {
        slog.Error("unable to invoke request",
            "err", err,
        )
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        slog.Error("unable to read response body",
            "err", err,
        )
    }

    slog.Info(string(body))
}

And:

ENDPOINT="..."

go run . --endpoint=${ENDPOINT}

And:

curl \
--silent ${ENDPOINT}/api/v2/alerts \
| jq -r .

[
  {
    "annotations": {
      "summary": "This is a test alert."
    },
    "endsAt": "2024-04-12T15:30:41.521-07:00",
    "fingerprint": "a6fe2270a4e28dea",
    "receivers": [
      {
        "name": "web.hook"
      }
    ],
    "startsAt": "2024-04-11T15:30:41.521-07:00",
    "status": {
      "inhibitedBy": [],
      "silencedBy": [],
      "state": "active"
    },
    "updatedAt": "2024-04-11T22:30:44.446Z",
    "generatorURL": "http://foo/test",
    "labels": {
      "alertname": "testAlert",
      "severity": "critical"
    }
  }
]