Reputation: 4601
I am newbie for using monitoring tools such as Prometheus in k8s..We have two separate cluster one for applications we deployed one for we only would like deploy monitoring,logging tools.
But I have some confusion how to handle this?
1.How cluster that serves prometheus can connect to application cluster and able to pull metrics?
2.How should I specify the namespace if I would like to set a network policy?
3.What should I do in application side for helm chart except exporting metrics?
# Allow traffic from pods with label app=prometheus in namespace with label name=monitoring
# to any pod in <YOUR_APPLICATION_NAMESPACE>
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: monitoring.prometheus.all
namespace: <YOUR_APPLICATION_NAMESPACE>
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
name: monitoring
podSelector:
matchLabels:
app: prometheus
podSelector: {}
policyTypes:
- Ingress
Upvotes: 0
Views: 2048
Reputation: 8411
Isn't that what you want?
Prometheus federation is a Prometheus server that can scrape data from other Prometheus servers. It supports hierarchical federation, which in our case resembles a tree.
A default version of the Prometheus server is installed in each one of our clusters and a Prometheus federation server is deployed together with Grafana in a central monitoring cluster. Prometheus federation scrapes data from all the other Prometheus servers that run in our clusters. For future expansion, a central Prometheus federation can be used to scrape data from multiple Prometheus federation servers that scrape data from groups of tens of clusters.
More info here: https://developers.mattermost.com/blog/cloud-monitoring/
2) Prometheus configuration to scrape Kubernetes outside the cluster yaml example
3) Linkedin Monitoring Kubernetes with Prometheus - outside the cluster! article and Reddit Monitoring K8s by Prometheus Outside Cluster related discussion
Upvotes: 1