Dinesh Kumar
Dinesh Kumar

Reputation: 493

Multiple ingress controller is not working

I'm creating multiple ingress controller in different namespaces. Initially, it's creating a load balancer in AWS and attached pod IP addresses to target groups. After some days it is not updating the new pod IP to the target group. I've attached the ingress controller logs here.

E0712 15:02:30.516295       1 leaderelection.go:270] error retrieving resource lock namespace1/ingress-controller-leader-alb: configmaps "ingress-controller-le │
│ ader-alb" is forbidden: User "system:serviceaccount:namespace1:fc-serviceaccount-icalb" cannot get resource "configmaps" in API group "" in the namespace "namespace1"

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "fc-ingress"
  annotations:
    kubernetes.io/ingress.class: alb-namespace1
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets:
    alb.ingress.kubernetes.io/certificate-arn: 
    alb.ingress.kubernetes.io/ssl-policy:
    alb.ingress.kubernetes.io/security-groups:
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-path: '/'
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds:  '2'
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '5'
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/healthy-threshold-count: '5'
    alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
    alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=false
    alb.ingress.kubernetes.io/load-balancer-attributes: deletion_protection.enabled=false
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
    alb.ingress.kubernetes.io/target-group-attributes: slow_start.duration_seconds=0
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=300
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=false
  labels:
    app: fc-label-app-ingress
spec:
  rules:
     - host: "hostname1.com"
       http:
        paths:
        - backend:
            serviceName: service1
            servicePort: 80

     - host: "hostname2.com"
       http:
        paths:
        - backend:
            serviceName: service2
            servicePort: 80

     - host: "hostname3.com"
       http:
         paths:
         - backend:
             serviceName: service3
             servicePort: 80


ingress_controller.yaml

# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: fc-label-app-icalb
  name: fc-ingress-controller-alb
  namespace: namespace1
  # Namespace the ALB Ingress Controller should run in. Does not impact which
  # namespaces it's able to resolve ingress resource for. For limiting ingress
  # namespace scope, see --watch-namespace.
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fc-label-app-icalb
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: fc-label-app-icalb
    spec:
      containers:
        - args:
            # Limit the namespace where this ALB Ingress Controller deployment will
            # resolve ingress resources. If left commented, all namespaces are used.
            - --watch-namespace=namespace1

            # Setting the ingress-class flag below ensures that only ingress resources with the
            # annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
            # choose any class you'd like for this controller to respect.
            - --ingress-class=alb-namespace1

            # Name of your cluster. Used when naming resources created
            # by the ALB Ingress Controller, providing distinction between
            # clusters.
            - --cluster-name=$EKS_CLUSTER_NAME

          # AWS VPC ID this ingress controller will use to create AWS resources.
            # If unspecified, it will be discovered from ec2metadata.
            # - --aws-vpc-id=vpc-xxxxxx

            # AWS region this ingress controller will operate in.
            # If unspecified, it will be discovered from ec2metadata.
            # List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
            # - --aws-region=us-west-1

            # Enables logging on all outbound requests sent to the AWS API.
            # If logging is desired, set to true.
            # - ---aws-api-debug
            # Maximum number of times to retry the aws calls.
            # defaults to 10.
            # - --aws-max-retries=10
          env:
            # AWS key id for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_ACCESS_KEY_ID
            #  value: KEYVALUE

            # AWS key secret for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_SECRET_ACCESS_KEY
            #  value: SECRETVALUE
          # Repository location of the ALB Ingress Controller.
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.4
          imagePullPolicy: Always
          name: server
          resources: {}
          terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      serviceAccountName: fc-serviceaccount-icalb




---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: fc-label-app-icalb
  name: fc-clusterrole-icalb
rules:
  - apiGroups:
      - ""
      - extensions
    resources:
      - configmaps
      - endpoints
      - events
      - ingresses
      - ingresses/status
      - services
    verbs:
      - create
      - get
      - list
      - update
      - watch
      - patch
  - apiGroups:
      - ""
      - extensions
    resources:
      - nodes
      - pods
      - secrets
      - services
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: fc-label-app-icalb
  name: fc-clusterrolebinding-icalb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fc-clusterrole-icalb
subjects:
  - kind: ServiceAccount
    name: fc-serviceaccount-icalb
    namespace: namespace1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: fc-label-app-icalb
  name: fc-serviceaccount-icalb
  namespace: namespace1

Upvotes: 5

Views: 5104

Answers (1)

Karen Danielyan
Karen Danielyan

Reputation: 1960

I have had an issue like that on AKS. I have two Nginx Ingress Controllers:

  1. external-nginx-ingress
  2. internal-nginx-ingress

Only one worked at a time, Internal or external.

After specifying a unique election-id for each one the problem was fixed.

I use the following HELM chart:

Repository    = "https://kubernetes.github.io/ingress-nginx"
Chart         = "ingress-nginx"
Chart_version = "4.1.3"
K8s Version   = "1.22.4"

Deployment

kubectl get deploy -n ingress
NAME                                READY   UP-TO-DATE   AVAILABLE
external-nginx-ingress-controller   3/3     3            3           
internal-nginx-ingress-controller   1/1     1            1           

IngressClass

kubectl get ingressclass
NAME             CONTROLLER                      PARAMETERS
external-nginx   k8s.io/ingress-nginx            <none>
internal-nginx   k8s.io/internal-ingress-nginx   <none>
   

Deployment for External

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-nginx-ingress-controller
  namespace: ingress
  annotations:
    meta.helm.sh/release-name: external-nginx-ingress
    meta.helm.sh/release-namespace: ingress
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: external-nginx-ingress
      app.kubernetes.io/name: ingress-nginx
  template:
    spec:
      containers:
        - name: ingress-nginx-external-controller
          image: >-
            k8s.gcr.io/ingress-nginx/controller:v1.2.1
          args:
            - /nginx-ingress-controller
            - >-
              --publish-service=$(POD_NAMESPACE)/external-nginx-ingress-controller
            - '--election-id=external-ingress-controller-leader'
            - '--controller-class=k8s.io/ingress-nginx'
            - '--ingress-class=external-nginx'
            - '--ingress-class-by-name=true'

Deployment for Internal

apiVersion: apps/v1
kind: Deployment
metadata:
  name: internal-nginx-ingress-controller
  namespace: ingress
  annotations:
    meta.helm.sh/release-name: internal-nginx-ingress
    meta.helm.sh/release-namespace: ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: internal-nginx-ingress
      app.kubernetes.io/name: ingress-nginx
  template:
    spec:
      containers:
        - name: ingress-nginx-internal-controller
          image: >-
            k8s.gcr.io/ingress-nginx/controller:v1.2.1
          args:
            - /nginx-ingress-controller
            - >-
              --publish-service=$(POD_NAMESPACE)/internal-nginx-ingress-controller
            - '--election-id=internal-ingress-controller-leader'
            - '--controller-class=k8s.io/internal-ingress-nginx'
            - '--ingress-class=internal-nginx'
            - '--ingress-class-by-name=true'

Upvotes: 2

Related Questions