CODEWITHSUNDEEP
kubernetesmonitoringprometheus
Y Sun
Y Sun

Reputation: 1

A sample containerized application in Kubernetes unable to be shown as targets in Prometheus for scraping metrics

My goal is to reproduce the observations in this blog post: https://medium.com/kubernetes-tutorials/monitoring-your-kubernetes-deployments-with-prometheus-5665eda54045

So far I am able to deploy the example rpc-app applicaiton in my cluster, the following shows the two pods for this application is running:

NAMESPACE     NAME                                                   READY   STATUS    RESTARTS   AGE     IP            NODE                           NOMINATED NODE   READINESS GATES
default       rpc-app-deployment-64f456b65-5m7j5                     1/1     Running   0          3h23m   10.244.0.15   my-server-ip.company.com   <none>           <none>
default       rpc-app-deployment-64f456b65-9mnfd                     1/1     Running   0          3h23m   10.244.0.14   my-server-ip.company.com   <none>           <none>

The application exposes metrics and is confirmed by:

root@xxxxx:/u01/app/k8s # curl 10.244.0.14:8081/metrics

    # HELP go_gc_duration_seconds A summary of the GC invocation durations.
    # TYPE go_gc_duration_seconds summary
    go_gc_duration_seconds{quantile="0"} 0
    go_gc_duration_seconds{quantile="0.25"} 0
        ...

    rpc_durations_seconds{service="uniform",quantile="0.5"} 0.0001021102787270781
    rpc_durations_seconds{service="uniform",quantile="0.9"} 0.00018233200374804932
    rpc_durations_seconds{service="uniform",quantile="0.99"} 0.00019828258205623097
    rpc_durations_seconds_sum{service="uniform"} 6.817882693745326
    rpc_durations_seconds_count{service="uniform"} 68279

My prometheus pod is running in the same cluster. However I am unable to see any rpc_* meterics in the prometheus.

monitoring    prometheus-deployment-599bbd9457-pslwf                 1/1     Running   0          30m     10.244.0.21   my-server-ip.company.com   <none>           <none>

In the promethus GUI

click Status -> Servcie Discovery, I got

Service Discovery rpc-metrics (0 / 3 active targets)

click Status -> Targets show nothing (0 targets)

click Status -> Configuration The content can be seen as: https://gist.github.com/denissun/14835468be3dbef7bc924032767b9d7f

I am really new to Prometheus/Kubernetes monitoring, appreciate your help to troubleshoot this issue.

update 1 - I created the service

`
# cat rpc-app-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: rpc-app-service
  labels:
    app: rpc-app
spec:
  ports:
  - name: web
    port: 8081
    targetPort: 8081
    protocol: TCP
    nodePort: 32325
  selector:
    app: rpc-app
  type: NodePort


# kubectl get service rpc-app-service
NAME              TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
rpc-app-service   NodePort   10.110.204.119   <none>        8081:32325/TCP   9h

Upvotes: 0

Views: 94

Answers (1)

DazWilkin
DazWilkin

Reputation: 40296

Did you create the Kubernetes Service to expose the Deployment?

kubectl create -f rpc-app-service.yam

The Prometheus configuration watches for Service endpoints not Deployments|Pods.

Have a look at the Prometheus Operator. It's slightly more involved than running a Prometheus Deployment in your cluster but it represents a state-of-the-art deployment of Prometheus with some elegant abstractions such as PodMonitors and ServiceMonitors.

Upvotes: 0

Related Questions