cant create a second ingress controller using helm with custom class in Azure k8s cluster

Question

I have created an ingress controller using Helm with default configuration

default        nginx-ingress-controller        LoadBalancer   10.0.182.128   xx.xxx.xx.90     80:32485/TCP,443:31756/TCP   62m
default        nginx-ingress-default-backend   ClusterIP      10.0.12.39     <none>           80/TCP                       62m

using Helm:

helm install nginx-ingress stable/nginx-ingress \         
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="Created static IP" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="XXX-aks-ingress"

this ingress is running in the default namespace.

Now, I wanted to add a second ingress controller, from the official doc I have specific Ingress class

helm install nginx-ingress stable/nginx-ingress \     
--namespace ingress-nginx-devices \ #I create this namespace first 
--set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources  
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="A second static Ip address created before" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"

but I keep getting this error:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "nginx-ingress" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "ingress-nginx-devices": current value is "default"

What could be wrong here ? Any help is appreciated :)

Answer 1

Update :

Controller-value of the controller that is processing this ingressClass

With recent update you might have to use the controller.ingressClassByName

Official doc : https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx#values

helm install nginx-ingress-devices stable/nginx-ingress \     
    --namespace ingress-nginx-devices \ #I create this namespace first 
    --set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources  
    --set controller.ingressClassResource.name="nginx-devices" \
    --set controller.replicaCount=2 \     
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set controller.service.loadBalancerIP="A second static Ip address created before" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"

Old answer

you can try, what we are changing is name : nginx-ingress-devices instead of nginx-ingress

helm install nginx-ingress-devices stable/nginx-ingress \     
--namespace ingress-nginx-devices \ #I create this namespace first 
--set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources  
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="A second static Ip address created before" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"

error you are getting is due to already there is cluster role with same name : nginx-ingress due to that you are getting the error.

ClusterRoleBindings grant a user, group, or service account a ClusterRole’s power across the entire cluster.

You can get the reference file here : https://github.com/helm/charts/blob/master/stable/nginx-ingress/templates/clusterrole.yaml

Answer 2

With recent releases "controller.ingressClassResource.name" needs to get a different name when trying to deploy a second ingress.

As shown in the list of keys now there's a new value that you need to set:

Key	Type	Default	Description
...			...
controller.ingressClass	string	"nginx"	For backwards compatibility with ingress.class annotation, use ingressClass. Algorithm is as follows, first ingressClassName is considered, if not present, controller looks for ingress.class annotation
controller.ingressClassByName	bool	false	Process IngressClass per name (additionally as per spec.controller).
controller.ingressClassResource.controllerValue	string	"k8s.io/ingress-nginx"	Controller-value of the controller that is processing this ingressClass
controller.ingressClassResource.default	bool	false	Is this the default ingressClass for the cluster
controller.ingressClassResource.enabled	bool	true	Is this ingressClass enabled or not
controller.ingressClassResource.name	string	"nginx"	Name of the ingressClass

So, for example, for creating a secondary ingress for private access you can run this Helm command:

helm install nginx-ingress-private ingress-nginx/ingress-nginx \
 --create-namespace \
 --namespace ingress-private \
 --set controller.ingressClass="nginx-private" \
 --set controller.ingressClassResource.name="nginx-private" \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"="true"

Answer 3

For my case the issue was with ingressclass existed already. I have simply deleted the ingresclass and it worked like a charm.

# kubectl get ingressclass --all-namespaces

This will return the name of already existing ingressclass. For my case, it was nginx. Delete that ingressclass.

# kubectl delete ingressclass nginx --all-namespaces

Verify that ingressclass is deleted

# kubectl get ingressclass --all-namespaces
No resources found

Rerun the helm update command should work.

You can also run multiple ingrss controller in parallel with new ingressClassName & ingressClassResourceName. First of all get list of all existing class name.

kubectl get ingressclass --all-namespaces
NAME        CONTROLLER             PARAMETERS   AGE
nginx       k8s.io/ingress-nginx   <none>       203d

Create a new ingress controller with unique className with this command.

helm install nginx-ingress stable/nginx-ingress \     
--namespace ingress-nginx-devices \ #I create this namespace first 
--set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources 
--set controller.ingressClassResource.name="nginx-devices" # custom classResourceName   
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="A second static Ip address created before" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"