Openshift Prometheus Operator

Description

I have installed the Prometheus Operator provided by CoreOS in our OpenShift Cluster, and while the installation and setup seemed successful, I am unable to see any metrics, targets, service discovery, or other expected information in the Prometheus UI.

Environment

OpenShift Cluster Version: 4.13
Kubernetes Version: 1.26.7
Installation Method: Operator

Configuration Files I have configured the Prometheus Operator, ServiceMonitor, PodMonitor, Deployment for an example application, and a corresponding Service as follows:

Prometheus Configuration:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  creationTimestamp: '2024-02-05T09:22:56Z'
  generation: 8
  managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:availableReplicas': {}
          'f:conditions':
            .: {}
            'k:{"type":"Available"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Reconciled"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
          'f:paused': {}
          'f:replicas': {}
          'f:shardStatuses':
            .: {}
            'k:{"shardID":"0"}':
              .: {}
              'f:availableReplicas': {}
              'f:replicas': {}
              'f:shardID': {}
              'f:unavailableReplicas': {}
              'f:updatedReplicas': {}
          'f:unavailableReplicas': {}
          'f:updatedReplicas': {}
      manager: PrometheusOperator
      operation: Update
      subresource: status
      time: '2024-02-05T14:31:36Z'
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          .: {}
          'f:evaluationInterval': {}
          'f:portName': {}
          'f:scrapeInterval': {}
          'f:serviceMonitorNamespaceSelector': {}
          'f:serviceMonitorSelector': {}
      manager: Mozilla
      operation: Update
      time: '2024-02-05T15:03:29Z'
  name: prometheus
  namespace: dev
  resourceVersion: '75884116'
  uid: 6865b7a4-c369-4c24-a4f1-68e7d38f85bd
spec:
  evaluationInterval: 30s
  portName: web
  scrapeInterval: 30s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: '2024-02-05T14:31:36Z'
      status: 'True'
      type: Available
    - lastTransitionTime: '2024-02-05T14:31:04Z'
      status: 'True'
      type: Reconciled
  paused: false
  replicas: 1
  shardStatuses:
    - availableReplicas: 1
      replicas: 1
      shardID: '0'
      unavailableReplicas: 0
      updatedReplicas: 1
  unavailableReplicas: 0
  updatedReplicas: 1

Service Monitor Configuration:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"labels":{"team":"frontend"},"name":"example-app","namespace":"dev"},"spec":{"endpoints":[{"port":"web"}],"selector":{"matchLabels":{"app":"example-app"}}}}
  resourceVersion: '75824311'
  name: example-app
  uid: 6fa8f7c4-7555-44ee-a500-246013501cfe
  creationTimestamp: '2024-02-05T13:55:15Z'
  generation: 1
  managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:team': {}
        'f:spec':
          .: {}
          'f:endpoints': {}
          'f:selector': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-02-05T13:55:15Z'
  namespace: dev
  labels:
    team: frontend
spec:
  endpoints:
    - port: web
  selector:
    matchLabels:
      app: example-app

Pod Monitor Configuration:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"monitoring.coreos.com/v1","kind":"PodMonitor","metadata":{"annotations":{},"labels":{"team":"frontend"},"name":"example-app","namespace":"dev"},"spec":{"podMetricsEndpoints":[{"port":"web"}],"selector":{"matchLabels":{"app":"example-app"}}}}
  resourceVersion: '75824417'
  name: example-app
  uid: fe97f130-d990-4ddc-8d5c-84fb5398156a
  creationTimestamp: '2024-02-05T13:55:21Z'
  generation: 1
  managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:team': {}
        'f:spec':
          .: {}
          'f:podMetricsEndpoints': {}
          'f:selector': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-02-05T13:55:21Z'
  namespace: dev
  labels:
    team: frontend
spec:
  podMetricsEndpoints:
    - port: web
  selector:
    matchLabels:
      app: example-app

Example App Deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: example-app
  namespace: dev
  uid: 1393addd-e918-4bd7-a7fe-a6b31732c25a
  resourceVersion: '75824344'
  generation: 1
  creationTimestamp: '2024-02-05T13:54:57Z'
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"example-app","namespace":"dev"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"example-app"}},"template":{"metadata":{"labels":{"app":"example-app"}},"spec":{"containers":[{"image":"fabxc/instrumented_app","name":"example-app","ports":[{"containerPort":8080,"name":"web"}]}]}}}}
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: apps/v1
      time: '2024-02-05T13:54:57Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
        'f:spec':
          'f:progressDeadlineSeconds': {}
          'f:replicas': {}
          'f:revisionHistoryLimit': {}
          'f:selector': {}
          'f:strategy':
            'f:rollingUpdate':
              .: {}
              'f:maxSurge': {}
              'f:maxUnavailable': {}
            'f:type': {}
          'f:template':
            'f:metadata':
              'f:labels':
                .: {}
                'f:app': {}
            'f:spec':
              'f:containers':
                'k:{"name":"example-app"}':
                  .: {}
                  'f:image': {}
                  'f:imagePullPolicy': {}
                  'f:name': {}
                  'f:ports':
                    .: {}
                    'k:{"containerPort":8080,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                  'f:resources': {}
                  'f:terminationMessagePath': {}
                  'f:terminationMessagePolicy': {}
              'f:dnsPolicy': {}
              'f:restartPolicy': {}
              'f:schedulerName': {}
              'f:securityContext': {}
              'f:terminationGracePeriodSeconds': {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2024-02-05T13:55:16Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:deployment.kubernetes.io/revision': {}
        'f:status':
          'f:availableReplicas': {}
          'f:conditions':
            .: {}
            'k:{"type":"Available"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:lastUpdateTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Progressing"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:lastUpdateTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
          'f:observedGeneration': {}
          'f:readyReplicas': {}
          'f:replicas': {}
          'f:updatedReplicas': {}
      subresource: status
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: example-app
    spec:
      containers:
        - name: example-app
          image: fabxc/instrumented_app
          ports:
            - name: web
              containerPort: 8080
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
status:
  observedGeneration: 1
  replicas: 3
  updatedReplicas: 3
  readyReplicas: 3
  availableReplicas: 3
  conditions:
    - type: Available
      status: 'True'
      lastUpdateTime: '2024-02-05T13:55:16Z'
      lastTransitionTime: '2024-02-05T13:55:16Z'
      reason: MinimumReplicasAvailable
      message: Deployment has minimum availability.
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2024-02-05T13:55:16Z'
      lastTransitionTime: '2024-02-05T13:54:57Z'
      reason: NewReplicaSetAvailable
      message: ReplicaSet "example-app-796f6d7c65" has successfully progressed.

Example App Service:

kind: Service
apiVersion: v1
metadata:
  name: example-app
  namespace: dev
  uid: a8ced819-804c-40a4-9ea9-8df331c034e3
  resourceVersion: '75824214'
  creationTimestamp: '2024-02-05T13:55:09Z'
  labels:
    app: example-app
    namespace: dev
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"example-app","namespace":"dev"},"name":"example-app","namespace":"dev"},"spec":{"ports":[{"name":"web","port":8080}],"selector":{"app":"example-app"}}}
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: v1
      time: '2024-02-05T13:55:09Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:app': {}
            'f:namespace': {}
        'f:spec':
          'f:internalTrafficPolicy': {}
          'f:ports':
            .: {}
            'k:{"port":8080,"protocol":"TCP"}':
              .: {}
              'f:name': {}
              'f:port': {}
              'f:protocol': {}
              'f:targetPort': {}
          'f:selector': {}
          'f:sessionAffinity': {}
          'f:type': {}
spec:
  clusterIP: 172.30.94.104
  ipFamilies:
    - IPv4
  ports:
    - name: web
      protocol: TCP
      port: 8080
      targetPort: 8080
  internalTrafficPolicy: Cluster
  clusterIPs:
    - 172.30.94.104
  type: ClusterIP
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    app: example-app
status:
  loadBalancer: {}

Network Policy:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-all
  namespace: dev
  uid: 41e45d6f-9226-4659-9999-2443984bdf96
  resourceVersion: '75676249'
  generation: 1
  creationTimestamp: '2024-02-05T11:05:43Z'
  managedFields:
    - manager: Mozilla
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-02-05T11:05:43Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:egress': {}
          'f:ingress': {}
          'f:policyTypes': {}
spec:
  podSelector: {}
  ingress:
    - {}
  egress:
    - {}
  policyTypes:
    - Ingress
    - Egress
status: {}

The Prometheus configuration can be found under Status > Configurations (via the Prometheus UI):

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  evaluation_interval: 30s
  external_labels:
    prometheus: dev/prometheus
    prometheus_replica: prometheus-prometheus-0
scrape_configs:
- job_name: serviceMonitor/dev/example-app/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (example-app);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - dev
- job_name: serviceMonitor/dev/prometheus-exporter-operator-controller-manager-metrics-monitor/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_control_plane, __meta_kubernetes_service_labelpresent_control_plane]
    separator: ;
    regex: (controller-manager);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - dev
- job_name: serviceMonitor/dev/prometheus-operator/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
    separator: ;
    regex: (prometheus-operator);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - dev

Upvotes: 1

Answers (3)

WJS

Reputation: 40024

You shouldn't be using a map to do that. Putting the values in a class and then using a Map to hold each instance of the class would be better.

read input from console, each row is one line.
Split the string using \\s+ for one or more chars of white space.
concatenate the first and last name as the key to the map.
pass the fields to the constructor to create an instance of the Person class



Map<String, Person> map = new HashMap<>();
Scanner scanner = new Scanner(System.in);
System.out.print("Enter number of rows: ");
int nRows = scanner.nextInt();
System.out.println("Enter + " nRows + " of data.");
for (int i = 0; i < nRows; i++) {
    String[] fields = row.split("\\s+");
    map.put(fields[0] + fields[1], new Person(fields[0],
            fields[1], fields[2], fields[3]));
}

map.entrySet().forEach(System.out::println);

prints

name1lastname1=[name1 lastname1, card1, country1]
name3lastname3=[name3 lastname3, card3, country3]
name2lastname2=[name2 lastname2, card2, country2]

Here is the class definition. There are other possibilities here to like changing the toString() return and adding setters.

class Person {
    private String firstName;
    private String lastName;
    private String card;
    private String country;
    
    public Person(String firstName, String lastName, String card,
            String country) {
        this.firstName = firstName;
        this.lastName = lastName;
        this.card = card;
        this.country = country;
    }
    
    public String getFirstName() {
        return firstName;
    }
    
    public String getLastName() {
        return lastName;
    }
    
    public String getCard() {
        return card;
    }
    
    public String getCountry() {
        return country;
    }
    
    @Override
    public String toString() {
        return "[" + firstName + " " + lastName + ", " + card + ", "
                + country + "]";
    }
}

You could also use streams to create the map but in this case there is no real advantage. And I would recommend you get acquainted with more basic aspects of Java before venturing into using streams.

Upvotes: 0

Sai H.

Reputation: 66

Instead of using a HashMap, it is a better idea to create a custom class with String name, lastName, creditCard, country as your variables. Each instance of the class will be like a set of data, and they can be loaded into an ArrayList or inserted into a database such as SQLite. I am not sure why you need to use a HashMap, is there a particular reason for your choice?

You might find this YouTube video useful

Upvotes: 0