Grafana alert provisioning issue

Question

I'd like to be able to provision alerts, and try to follow this instructions, but no luck!

OS Grafana version: 9.2.0 Running in Kubernetes

Steps that I take:

Created a new alert rule from UI.
Extract alert rule from API: curl -k https:///api/v1/provisioning/alert-rules/-4pMuQFVk -u admin:

It returns the following:

---
id: 18
uid: "-4pMuQFVk"
orgID: 1
folderUID: 3i72aQKVk
ruleGroup: cpu_alert_group
title: my_cpu_alert
condition: B
data:
- refId: A
  queryType: ''
  relativeTimeRange:
    from: 600
    to: 0
  datasourceUid: _SaubQF4k
  model:
    editorMode: code
    expr: system_cpu_usage
    hide: false
    intervalMs: 1000
    legendFormat: __auto
    maxDataPoints: 43200
    range: true
    refId: A
- refId: B
  queryType: ''
  relativeTimeRange:
    from: 0
    to: 0
  datasourceUid: "-100"
  model:
    conditions:
    - evaluator:
        params:
        - 3
        type: gt
      operator:
        type: and
      query:
        params:
        - A
      reducer:
        params: []
        type: last
      type: query
    datasource:
      type: __expr__
      uid: "-100"
    expression: A
    hide: false
    intervalMs: 1000
    maxDataPoints: 43200
    refId: B
    type: classic_conditions
updated: '2022-12-07T20:01:47Z'
noDataState: NoData
execErrState: Error
for: 5m

Deleted the alert rule from UI.
Made a configmap from above alert-rule, as such:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-alerting
data:
  alerting.yaml: |-
    apiVersion: 1
    groups:
      - id: 18
        uid: "-4pMuQFVk"
        orgID: 1
        folderUID: 3i72aQKVk
        ruleGroup: cpu_alert_group
        title: my_cpu_alert
        condition: B
        data:
        - refId: A
          queryType: ''
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: _SaubQF4k
          model:
            editorMode: code
            expr: system_cpu_usage
            hide: false
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: true
            refId: A
        - refId: B
          queryType: ''
          relativeTimeRange:
            from: 0
            to: 0
          datasourceUid: "-100"
          model:
            conditions:
            - evaluator:
                params:
                - 3
                type: gt
              operator:
                type: and
              query:
                params:
                - A
              reducer:
                params: []
                type: last
              type: query
            datasource:
              type: __expr__
              uid: "-100"
            expression: A
            hide: false
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: classic_conditions
        updated: '2022-12-07T20:01:47Z'
        noDataState: NoData
        execErrState: Error
        for: 5m

I mount above configmap in grafana container (in /etc/grafna/provisioning/alerting). The full manifest of the deployment is as follow:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2022-12-08T18:31:30Z"
  generation: 4
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 9.3.0
    helm.sh/chart: grafana-6.46.1
  name: grafana
  namespace: monitoring
  resourceVersion: "648617"
  uid: dc06b802-5281-4f31-a2b2-fef3cf53a70b
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: grafana
      app.kubernetes.io/name: grafana
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: 98cac51656714db48a116d3109994ee48c401b138bc8459540e1a497f994d197
        checksum/dashboards-json-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        checksum/sc-dashboard-provider-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        checksum/secret: 203f0e4d883461bdd41fe68515fc47f679722dc2fdda49b584209d1d288a5f07
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: grafana
        app.kubernetes.io/name: grafana
    spec:
      automountServiceAccountToken: true
      containers:
      - env:
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: admin-user
              name: grafana
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: admin-password
              name: grafana
        - name: GF_PATHS_DATA
          value: /var/lib/grafana/
        - name: GF_PATHS_LOGS
          value: /var/log/grafana
        - name: GF_PATHS_PLUGINS
          value: /var/lib/grafana/plugins
        - name: GF_PATHS_PROVISIONING
          value: /etc/grafana/provisioning
        image: grafana/grafana:9.3.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 10
          httpGet:
            path: /api/health
            port: 3000
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        name: grafana
        ports:
        - containerPort: 3000
          name: grafana
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/health
            port: 3000
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/grafana/grafana.ini
          name: config
          subPath: grafana.ini
        - mountPath: /etc/grafana/provisioning/alerting
          name: grafana-alerting
        - mountPath: /var/lib/grafana
          name: storage
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 472
        runAsGroup: 472
        runAsUser: 472
      serviceAccount: grafana
      serviceAccountName: grafana
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: grafana-alerting
        name: grafana-alerting
      - configMap:
          defaultMode: 420
          name: grafana
        name: config
      - emptyDir: {}
        name: storage

However, grafana fail to start with this error:

Failed to start grafana. error: failure to map file alerting.yaml: failure parsing rules: rule group has no name set

I fixed above error by adding Group Name, but similar errors about other missing elements kept showing up again and again (to the point that I stopped chasing, as I couldn't figure out what exactly the correct schema is). Digging in, it looks like the format/schema returned from API in step 2, is different than the schema that's pointed out in the documentation.

Why is the schema of the alert-rule returned from API different? Am I supposed to convert it, and if so how? Otherwise, what am I doing wrong?

Edit: Replaced Statefulset with deployment, since I was able to reproduce this in a normal/minimal Grafana deployment too.

Grafana alert provisioning issue

Answers (1)

Related Questions