CODEWITHSUNDEEP
kubernetesprometheusprometheus-operatorservicemonitor
mona moghadampanah
mona moghadampanah

Reputation: 343

servicemonitor doesnot add to prometheus-operator targets

I have deployed a Prometheus-operator on the k8s cluster. Everything works well but I want to monitor MySQL pods that are in another namespace. I create mysqld-exporter pod and svc for it in MariaDB namespace and a servicemonitor for it in the monitoring namespace. I check all the items which are in this link, but this servicemonitor(for mysqld) doesn't add to Prometheus targets. when I change the svc type to nodeport everything works, and metrics are exposed.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2022-09-11T11:51:46Z"
  generation: 1
  labels:
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 9.1.2
    monitor-app: mysqld-exporter
  name: mysqld-exporter
  namespace: monitoring
  resourceVersion: "2932040"
  uid: 247683c8-7868-4f2c-9a60-255c703273a5
spec:
  endpoints:
  - interval: 30s
    port: http-metrics
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - mariadb
  selector:
    matchLabels:
      app: mysqld-exporter


--------

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-09-11T11:50:28Z"
  labels:
    app: mysqld-exporter
  name: mysqld-exporter
  namespace: mariadb
  resourceVersion: "2931235"
  uid: 1b548f89-33a1-4235-b042-8cda5dfc766b
spec:
  clusterIP: 10.109.39.231
  clusterIPs:
  - 10.109.39.231
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http-metrics
    port: 9104
    protocol: TCP
    targetPort: 9104
  selector:
    app: mysqld-exporter
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Upvotes: 0

Views: 1780

Answers (2)

salborough
salborough

Reputation: 29

I had the same issue and modifying the clusterRole permissions as per the above link https://github.com/prometheus-operator/kube-prometheus/issues/483#issuecomment-610427646 solved this issue for me. Now Prometheus can scrape from pods in another namespace.

Upvotes: -1

mona moghadampanah
mona moghadampanah

Reputation: 343

I checked the Prometheus pod logs, and its error was :

pods is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list resource

so I search for this error and find the answer in this link and add pods and services to the resources of Prometheus-k8s ClusterRole.

Upvotes: 1

Related Questions