Rabbitmq service not reachable within k3s pods?

Question

I have the following setup: k3s installation and istio as ingress controller,

problem we face that rabbitmq service is not reachable from within pods internally.

• A Service that has rabbitmq port exposed [5671] -> type: ClusterIP

   $ kubectl describe service rabbit-svc -n rabbit
   ......
   Type:              ClusterIP
   IP Family Policy:  SingleStack
   IP Families:       IPv4
   IP:                10.43.139.126
   IPs:               10.43.139.126
   ......
   Port:              https  443/TCP
   TargetPort:        443/TCP
   Endpoints:         10.42.0.222:443
   Port:              rabbitmq  5671/TCP
   TargetPort:        5671/TCP
   Endpoints:         10.42.0.222:5671
  

• if I try to connect to that service via clusterIP from inside the machine outside k3s pods, it responds back

  openssl s_client -connect 10.43.139.126:5671
  CONNECTED(00000003)
  

• if I try to connect to the pod from istio pod via the service k8s fqdn, it gives connection refused

  openssl s_client -connect rabbit-svc.rabbit.svc.cluster.local:5671
  139693269235008:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:110:
  139693269235008:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
  connect:errno=111
  

Any idea why this is happening though I have another port from inside that rabbitmq service exposed via 443 and it respond back fine [ that's just to rule out DNS issues ]

Answer 1

The connection error:111 is might be due to below concerns and can you check on this :

• Check whether the Port 5671 is open or blocked by using nmap or telnet.
• Check if there are any firewalls blocking this port/host. Try to disable the firewall and have a check
• Seems to be your istio is having SSL certificate enabled - this blocking the port 5671 to reach the connection. Can you disable the SSL certificate and have a check. If it is connected then SSL certificate is the reason which is blocking this. In this case you need to make sure 5671 Port should accept the SSL certificate. If there is no SSL certificate enabled then the first two points might be the reason.
• Can you try by using this command echo | openssl s_client -connect <hostname>:<port> | head

Refer to this RabbitMQ troubleshooting doc for more information