Intermittent ingress nginx upstream timed out (110: Operation timed out) while connecting to upstream

Question

This is NOT "while reading resp*nse header from upstream" and the issue is NOT a slow response from the upstream server.

I did this curl: curl https://knotfree.net/api1/getGiantPassword

What I get is a 5 second delay. What I expect is no delay, the same as this curl: curl http://knotfree.io/api1/getGiantPassword

If I restart the ingress-nginx pod then the first request completes normally. After that it goes like this. I curl:

curl https://knotfree.net/api1/getGiantPassword

Then nginx wait for 5 long lonely seconds before coming to the conclusion:

2024/07/06 18:35:06 [error] 26#26: *556202 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.3.96.7, server: knotfree.net, request: "GET /api1/getGiantPassword HTTP/2.0", upstream: "http://[fd10:1ba:6d2c:1000:3a5b:26d5:2b5f:56b6]:8085/api1/getGiantPassword", host: "knotfree.net"

And then the fun part. It sends the request to the upstream; which logs the request; and then replies in a millisecond; and then nginx logs:

10.3.96.7 - - [06/Jul/2024:18:45:10 +0000] "GET /api1/getGiantPassword HTTP/2.0" 200 79 "-" "curl/8.6.0" 46 5.002 [knotspace-knotfreeaide-80] [] [fd10:1ba:6d2c:1000:3a5b:26d5:2b5f:56b6]:8085, 10.244.183.182:8085 0, 79 5.000, 0.002 504, 200 6e1e1ab19b8964722f112dc49e642238

Which is a 200 and curl shows the correct output (basically a random number).

How is nginx both having an error and also serving the correct result? Why does it sometimes work perfectly? Note that the ingress-nginx controller is doing the https termination. My other, non-secure endpoint works perfectly always:

curl http://knotfree.io/api1/getGiantPassword

What does it mean that it can't connect with upstream? Am I supposed to be serving a health check? This is on vultr kubernetes.

FYI. Here's the Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    external-dns.alpha.kubernetes.io/hostname: "knotfree.net"  
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    certmanager.k8s.io/issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: dns01
    certmanager.k8s.io/acme-dns01-provider: vultr
    
  name: nginx-ingress
spec:
  rules:
  - host: "knotfree.net" 
    http:
      paths:
      - backend:
          service:
            name: knotfreeaide
            port:
              number: 80
        path: /
        pathType: Prefix
  - host: "*.knotfree.net" 
    http:
      paths:
      - backend:
          service:
            name: knotfreeaide
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - "knotfree.net"
    - "*.knotfree.net"
    secretName: wildcard-tls