CODEWITHSUNDEEP
prometheuskubernetes-helmkustomize
Michael Boiman
Michael Boiman

Reputation: 41

Deploy kube-prometheus-stack with Kustomize don't work

I am trying to deploy kube-prometheus-stack https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack through kustomize. But I am getting from the deployment:

unable to recognize ".": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1"
unable to recognize ".": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"
unable to recognize ".": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
...(and so on)

Full output under: https://app.warp.dev/block/JJwOYMJZng9CyBdVlBaIIF

I tried to deploy local on rancher desktop but on docker desktop I get the same.

What I did: There are nor kustomize file for this stack so I take the manifest with:

helm template prometheus-community/kube-prometheus-stack > prometheus.yaml

after this didn't work I tried to take directly in my kustomize.yaml the helm with:

 helmCharts:
 - name: kube-prometheus-stack 
   repo: https://prometheus-community.github.io/helm-charts
   version: 35.0.3
   releaseName: prometheus

and started kustomize with:

kubectl kustomize . --enable-helm | kubectl apply -f -

both have the same problem.

If use helm with:

helm install prometheus prometheus-community/kube-prometheus-stack

its working.

What interesting is that when I uninstall it:

helm uninstall prometheus

and then deploy it again through kustomize

kubectl apply -k .

Its working, but it is not the solution which I need. So what am I doing wrong?

Upvotes: 0

Views: 2992

Answers (3)

ckeeney
ckeeney

Reputation: 1386

I solved this by setting helmCharts.includeCRDs: true in kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: monitoring

helmCharts:
  - name: kube-prometheus-stack
    repo: https://prometheus-community.github.io/helm-charts
    version: 45.28.1
    includeCRDs: true
    releaseName: kube-prometheus-stack
    namespace: monitoring
    valuesFile: values.yaml

Upvotes: 0

Ghis
Ghis

Reputation: 923

I had a similar problem with Helmfile. When the CRDs are deployed at the same time (in the same chart or even as Helmfile dependency) as the resources, Helmfile validation will fail :

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

However, Kubernetes agent is perfectly capable of handling crd/resource creation at the same time. The solution with Helmfile was to disable validation on first deploy :

helmfile.yaml

releases:
- name: prometheus
  disableValidation: true # Missing CRDs on first deploy
  installed: true
  values:
    - values.yaml

It's the only workaround I have found so far, I'm not very familiar with Kustomize, but I suppose there must be a way to skip validation as well ?

Hope this helps anyway

Upvotes: 0

Alexander Meise
Alexander Meise

Reputation: 1438

You need to give kubernetes some time to create CRDs before trying to deploy objects that "use" those CRDs.

Next time just do the apply one time, wait 10 seconds and do the apply again. even several times until the errors disappear.

Upvotes: 1

Related Questions