Ertan Hasani
Ertan Hasani

Reputation: 1773

Https not working locally for ingress on my kubernetes cluster

I am trying to make my ingress run over https but I'm getting HTTP ERROR 403, and I don't know what am I missing here.

I am trying to access them on my-local-domain.com and I added that entry on my etc/hosts file as 127.0.0.1 my-local-domain.com so I can access it that way.

Here's my Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: cert-issuer
  namespace: my-namespace
spec:
  selfSigned: {}

Here's my Certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: self-signed-cert
  namespace: my-namespace
spec:
  secretName: my-secret-tls
  dnsNames:
    - my-local-domain.com
    - www.my-local-domain.com
  issuerRef:
    name: cert-issuer

While here is my nginx ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  labels:
      name: ingress-service
  namespace: my-namespace
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    namespace: my-namespace
spec:
  tls:
    - hosts:
      - my-local-domain.com
      - www.my-local-domain.com
      secretName: my-secret-tls
  rules:
  - host: my-local-domain.com
    http:
      paths:
      - path: /api/(.*)
        pathType: Prefix
        backend:
          service:
            name: web-api
            port:
              number: 443
      - path: /(.*)
        pathType: Prefix
        backend:
          service:
            name: web-client
            port:
              number: 80

nginx.conf for my web-client service

server {
        listen 80;

        location / {
            root /usr/share/nginx/html;
            index index.html index.htm;
            try_files $uri $uri/ /index.html;
        }

        gzip  on; 
    }

And when I try to access it with https://my-local-domain.com

Access to my-local-domain.com was denied
You don't have authorization to view this page.
HTTP ERROR 403

I also tried to run

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "//CN=my-local-domain.com\O=my-local-domain.com" -addext "subjectAltName = DNS:my-local-domain.com"

kubectl create secret tls tls-secret --key tls.key --cert tls.crt -n my-namespace

remove Certificate and CertificateIssuer and add tls-secret as secretName in ingress, as well as added those tls.key and tls.crt to my web-client nginx and updated it to:

    server {
      listen 443 ssl default_server;

      ssl_protocols SSLv3 TLSv1;

      server_name www.my-local-domain.com my-local-domain.com;

      access_log /var/log/nginx/access.log;
      error_log  /var/log/nginx/error.log info;

      keepalive_timeout 75 75;

      ssl_certificate /etc/ssl/certs/self-signed.crt;
      ssl_certificate_key /etc/ssl/private/self-signed.key;
      ssl_session_timeout  5m;

      add_header Strict-Transport-Security "max-age=7200";
      
      location / {
          root /usr/share/nginx/html;
          index index.html index.htm;
          try_files $uri $uri/ /index.html;
      }
    }

p.s. when doing this, i run everyrhing on port 443 instead of 80.

When I remove the tls part from my ingress, then it does work through HTTP. And I also tried to change my dns/host from my-local-domain.com to be only localhost but still the same. And even if I try to access https://localhost I get that HTTP ERROR 403.

Upvotes: 2

Views: 3291

Answers (1)

akop
akop

Reputation: 7835

I don't know if this will help, but here are some advices:

  1. Determine the source of your error
    You provide us a lot of code, but no details of the error. Is the 403 thrown by your webservice, your api or by your ingress-controller? You can check that in the error-page. Errors from ingress-controller usally contains somethink like "openresty" in footer. But it hardly depends on your ingress-controller which are you using in your k8s.
    You can also check the logs of your containers (nginx write the logs to /var/log/nginx/error.log). Here you can find some more detailed informations why you're getting a 403.

  2. Let the cluster handle SSL-termination
    Don't put the ssl-certifications into the containers. It will increase hardly the complexity of your app(s).
    It is a lot easier, when the cluster take care about that. So your app can be faster deployed on other clusters with others environments.

  3. Avoid the cert-manager, when using self-signed certs
    As you already tested, but I suggest to use the cert-manager only, when you need certificates from lets encrypt. Otherwise it will increase the complexity again.

  4. Don't mix up HTTP and HTTPS
    Your webapi using HTTPS for the internal communication (internal means: ingress-controller -> webapi-service, which is usually unencrypted) ... but not for your web-server.
    Keep things simple - HTTPS for all or for nobody.

  5. Try to avoid url-rewrite
    Whenever when possible, try to avoid url-rewrite. I see very often problems with it, so I suggest to avoid it.
    I know this is not always possible, but when all componets are developed by you ... why did you need it at all? Here is your ingress in "easy":

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
       name: ingress-service
       labels:
          name: ingress-service
       namespace: my-namespace
       annotations:
          nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    
    spec:
       tls:
          - hosts:
             - my-local-domain.com
             - www.my-local-domain.com
          secretName: my-secret-tls
       rules:
          - host: my-local-domain.com
            http:
               paths:
                  - path: /api/       
                    backend:
                       service:
                          name: web-api
                          port:
                             number: 80
                  - path: /
                    backend:
                       service:
                       name: web-client
                       port:
                          number: 80  
    
  6. Rootless is the best
    Not very important for your problem, but instead of the standard nginx-image, use the unprivileged nginx-image. If your service (in this case nginx) got hacked, then the hacker has the same privileges like nginx. This means, he can easily install tools into the nginx-container and do very bad things.
    If you're using the unprivileged image, than the hacker can ... almost nothing. He needs a second exploit to escalate his rights.

Good luck to find your error. We are happy, when you share your next steps with us.

Upvotes: 1

Related Questions