CODEWITHSUNDEEP
sslkuberneteskubernetes-ingresslets-encryptnginx-ingress
Beefcake
Beefcake

Reputation: 334

Kubernetes Ingress TLS not being created with headless service

What I'm trying to achieve

I'm trying to deploy an elixir (phoenix) application in a microk8s cluster namespace with TLS using let's encrypt. The cluster is hosted on an AWS EC2 instance.

The problem I'm facing

The TLS secret is not being created in the namespace and a 'default' one is created

The secrets after deploying both phoenix app and httpbin app:

me@me:~/Documents/kubernetes-test$ kubectl get secret -n production
NAME                           TYPE                                  DATA   AGE
default-token-jmgrg            kubernetes.io/service-account-token   3      20m
httpbin-tls                    kubernetes.io/tls                     2      81s

The domain is insecure, i.e the TLS is not working.

Logs from the ingress controller after applying the yml files:

W0106 17:26:36.967036       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate
W0106 17:26:46.445248       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate
W0106 17:26:49.779680       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate
I0106 17:26:56.431925       6 status.go:281] "updating Ingress status" namespace="production" ingress="phoenix-app-ingress" currentValue=[] newValue=[{IP:127.0.0.1 Hostname: Ports:[]}]
I0106 17:26:56.443405       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"phoenix-app-ingress", UID:"REDACTED", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1145907", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0106 17:26:56.443655       6 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "production/phoenix-app-tls" in local store
W0106 17:26:56.443781       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate

The description of the created ingress, note that here at the bottom it says Successfully created Certificate "phoenix-app-tls" but the secret does not exist:

me@me:~/Documents/kubernetes-test$ kubectl describe ing phoenix-app-ingress -n production
Name:             phoenix-app-ingress
Labels:           app=phoenix-app
Namespace:        production
Address:          127.0.0.1
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  phoenix-app-tls terminates phoenix.sub.mydomain.com
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  phoenix.sub.mydomain.com  
                                 /   phoenix-app-service-headless:8000 (REDACTED_IP:4000,REDACTED_IP:4000)
Annotations:                     cert-manager.io/cluster-issuer: letsencrypt
                                 nginx.ingress.kubernetes.io/cors-allow-credentials: true
                                 nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, OPTIONS
                                 nginx.ingress.kubernetes.io/cors-allow-origin: *
                                 nginx.ingress.kubernetes.io/enable-cors: true
Events:
  Type    Reason             Age                  From                      Message
  ----    ------             ----                 ----                      -------
  Normal  CreateCertificate  29m                  cert-manager              Successfully created Certificate "phoenix-app-tls"
  Normal  Sync               8m43s (x3 over 29m)  nginx-ingress-controller  Scheduled for sync

Resources

The deployment yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: phoenix-app
  labels:
    app: phoenix-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: phoenix-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: phoenix-app
    spec:
      containers:
      - name: phoenix-app
        image: REDACTED
        imagePullPolicy: Always
        command: ["./bin/hello", "start"]
        lifecycle:
          preStop:
            exec:
              command: ["./bin/hello", "stop"]
        ports:
        - containerPort: 4000
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        envFrom:
        - configMapRef:
            name: phoenix-app-config
        - secretRef:
            name: phoenix-app-secrets
      imagePullSecrets:
      - name: gitlab-pull-secret

The service yml:

apiVersion: v1
kind: Service
metadata:
  name: phoenix-app-service-headless
  labels:
    app: phoenix-app
spec:
  clusterIP: None
  selector:
    app: phoenix-app
  ports:
  - name: http
    port: 8000
    targetPort: 4000 # The exposed port by the phoenix app

Note: I removed my actual domain

The ingress yml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: phoenix-app-ingress
  labels:
    app: phoenix-app
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - "phoenix.sub.mydomain.com"
    secretName: phoenix-app-tls
  rules:
  - host: "phoenix.sub.mydomain.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: phoenix-app-service-headless
            port:
              number: 8000 # Same port as in service.yml

Tested with different service

I deployed a sample service using httpbin (is not a headless service) and the TLS works fine in the same namespace. Here are the resources that I used to deploy it:

deplyoment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: Always
        name: httpbin
        ports:
        - containerPort: 80

The service yml:

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin

The ingress yml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  labels:
    app: httpbin
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - "httpbin.sub.mydomain.com"
    secretName: httpbin-tls
  rules:
  - host: "httpbin.sub.mydomain.com" # This is a subdomain we want to route these requests to
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 8000

My best guess is that it has something to do with the fact that the service is headless, but I have no clue as to how I can resolve the issue.

Upvotes: 2

Views: 1578

Answers (1)

Beefcake
Beefcake

Reputation: 334

I found out that you can actually check for certificates with kubectl: kubectl get certificate -n production

The status of this certificate was READY = FALSE.

I checked the description: kubectl describe certificate <certificate_name> -n production

At the bottom it said: Too many certificates have been created in the last 164 hours for this exact domain.

I just changed the domain and voila! It works.

Upvotes: 2

Related Questions