Kubernetes nginx-ingress isnt exposing the services

Question

I'm trying to achieve following, I have installed Kube cluster (version 1.18) using kops on AWS EC2. It's up and running but now I want to expose some workloads to the internet to be available via subdomains but it isnt working as expected and I use ingress for that.

More details below:

Client Version: v1.18.3
Server Version: v1.18.3

helm version --short
Client: v2.17.0+ga690bad
Server: v2.17.0+ga690bad

Chart is below:

deploy.yaml:
    
        name: custom-smscs
        branch: k8s
        git: git@github.com:REDACTED
        
        image:
          repository: <account_id>.dkr.ecr.eu-west-1.amazonaws.com/custom_smscs
        
        service:
          port: 80
          internalPort: 80
          healthEndpoint: /health

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ .Chart.Name }}-deployment"
  labels:
    app: {{ .Values.name }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Values.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
    spec:
      containers:
      - name: {{ .Values.name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - containerPort: {{ .Values.service.internalPort }}
        env:
        - name: NODE_ENV
          value: {{ .Values.environment }}
        livenessProbe:
          httpGet:
            path: {{ .Values.service.healthEndpoint }}
            port: {{ .Values.service.internalPort }}
          initialDelaySeconds: 15
          timeoutSeconds: 10
          periodSeconds: {{ .Values.service.healthPeriod }}
          successThreshold: 1
          failureThreshold: {{ .Values.service.healthThreshold }}
        resources:
{{ toYaml .Values.resources | indent 12 }}

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: "{{ .Values.name }}-service"
  labels:
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
spec:
  type: {{ .Values.service.type }}
  selector:
          app: "{{ .Values.name }}"
  ports:
  - port: {{ .Values.service.port }}
    targetPort: {{ .Values.service.internalPort }}
    protocol: TCP
    name: http
  {{- if .Values.service.ports }}
  {{- range $name, $port := .Values.service.ports }}
  - port: {{ $port }}
    targetPort: {{ $port }}
    protocol: TCP
    name: {{ $name }}
  {{- end }}
  {{- end }}
  selector:
    app: {{ .Values.name }}

ingress.yaml:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: {{ .Values.name }}-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: customsmscs.my-domain.com
    http:
      paths:
        - path: /
          backend:
            serviceName: {{ .Values.name }}-service
            servicePort: {{ .Values.service.port }}

values.yaml:

replicaCount: 1
image:
  repository: replaced
  tag: "0.0.1"
  pullPolicy: Always
service:
  type: ClusterIP
  name: replaced
  port: 80
  internalPort: 80
  healthEndpoint: /health
  healthPeriod: 30
  healthThreshold: 3
environment: production
resources:
  limits:
    cpu: 256m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  target: 80

The nginx was installed using following command:

kc apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/aws/deploy-tls-termination.yaml

There is everything required to run the nginx-ingress correctly:

    kc -n ingress-nginx get po
    NAME                                       READY   STATUS      RESTARTS   AGE
    ingress-nginx-admission-create-qltch       0/1     Completed   0          154m
    ingress-nginx-admission-patch-6ll84        0/1     Completed   0          154m
    ingress-nginx-controller-b4678dfcb-8gn56   1/1     Running     0          154m

    kc get all | grep custom
pod/custom-smscs-deployment-6cd5f58bb8-ht49w   1/1     Running   0          37m
service/custom-smscs-service   ClusterIP   100.70.5.207     <none>        80/TCP            37m
deployment.apps/custom-smscs-deployment   1/1     1            1           37m
replicaset.apps/custom-smscs-deployment-6cd5f58bb8   1         1         1       37m

    kc get ing
    NAME                   CLASS    HOSTS                                   ADDRESS                                                                   PORTS   AGE
   custom-smscs-ingress   <none>   customsmscs.my-domain.com   <ELB>.amazonaws.com   80      38m

    kc get ep
    NAME                   ENDPOINTS                               AGE
    custom-smscs-service   100.101.173.27:80                       40m

But when I do describe the ingress i noticed some error, not sure it's a reason or not:

nginx log:

    I0525 09:47:46.702227       6 controller.go:146] "Configuration changes detected, backend reload required"
    I0525 09:47:46.780938       6 controller.go:163] "Backend successfully reloaded"
    I0525 09:47:46.781209       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-b4678dfcb-8gn56", UID:"d5d9246b-63a8-4eba-8467-fdec0f3f9b3f", APIVersion:"v1", ResourceVersion:"89646", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
    I0525 09:48:19.528969       6 status.go:284] "updating Ingress status" namespace="default" ingress="custom-smscs-ingress" currentValue=[] newValue=[{IP: Hostname:<ELB>.eu-west-1.elb.amazonaws.com Ports:[]}]
    I0525 09:48:19.533791       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"custom-smscs-ingress", UID:"309890dd-1148-4aa8-bc14-62e57bbd969f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"267393", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

<error: endpoints "default-http-backend" not found>

        kc describe ing custom-smscs-ingress
    Name:             custom-smscs-ingress
    Namespace:        default
    Address:          <ELB>.eu-west-1.elb.amazonaws.com
    Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
    Rules:
      Host                                   Path  Backends
      ----                                   ----  --------
      customsmscs.my-domain.com  
                                             /   custom-smscs-service:80 (100.101.173.27:80)
    Annotations:                             kubernetes.io/ingress.class: nginx
    Events:
      Type    Reason  Age                From                      Message
      ----    ------  ----               ----                      -------
      Normal  Sync    43m (x2 over 43m)  nginx-ingress-controller  Scheduled for sync

So everything is looking good apart from the error above but when im trying to open an app.example.com in the browser im getting "Hmmm… can't reach this page". But when I open the ELB DNS endpoint URL in the browser I get "404 not found" which I guess is correct response as I called a default backend.

Another thing I noticed is that on the ELB there is only one of two nodes is in service:

I'm literally have spent 4 days on it but can't make it working.

Do you have any idea how to fix/overcome it guys?

Thank you in advance!

Answer 1

Alright, the solution was so easy.... I have added DNS A record (alias) like *.ks8.my-domain.com pointing to the ingress ELB and created an SSL certificate in AWS Certificate Manager, as well there is one change I had to do, in the ingress.yaml the *host: * parameter should be changed to customsmscs.k8s.my-domain.com instead of customsmscs.my-domain.com. And then I deleted the deployment and all related resources (ingress, service, etc) and redeployed it and now the app is available at the https://customsmscs.k8s.my-domain.com.

Thanks @thomas for advice, you helped me a lot!