CODEWITHSUNDEEP
kubernetestcprabbitmqistioistio-gateway
manu4543
manu4543

Reputation: 556

TCP proxy through Istio is not working in one of our cluster and working in other. What should I change in config? How do I debug further?

TCP proxy through Istio is not working in one of our cluster and working in other. Every configuration is same.

We have configured RabbitMQ and it is accepting TLS connection on 5671 port. If I port-forward the rabbitmq service and try to connect from localhost it works, but the same does not work through Istio TCP proxy. The same code works with RabbitMQ running in another cluster.

RabbitMQ logs shows no connection lifecycle related log. The connection closes abruptly with the following error.

error in connection to rabbitmq Error: Client network socket disconnected before secure TLS connection was established

Gateway and VS config

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: amqptls-ingressgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 5671
      name: ampqs
      protocol: TCP
    hosts:
    - "rabbitmq.xyz.io"

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: rabbitmq-virtual-service
  namespace: istio-system
spec:
  hosts:
  - "rabbitmq.xyz.io"
  gateways:
  - amqptls-ingressgateway
  tcp:
  - match:
    - port: 5671
    route:
    - destination:
        host: rabbitmq.xyz.svc.cluster.local
        port:
          number: 5671

Services

MacBook-Pro-3:Desktop manuchaudhary$k get svc -nistio-system
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                      AGE
infra-applications     ClusterIP   172.20.191.220   <none>        8080/TCP                                     284d
istio-ingressgateway   ClusterIP   172.20.142.243   <none>        15021/TCP,80/TCP,443/TCP,5671/TCP,5672/TCP   285d
istiod                 ClusterIP   172.20.55.27     <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP        285d

RabbitMQ

MacBook-Pro-3:Desktop manuchaudhary$ k get svc -nxyz | grep rabbitmq
rabbitmq                     ClusterIP   172.20.36.189    <none>        5672/TCP,5671/TCP,4369/TCP,25672/TCP,15672/TCP,9419/TCP   72m
rabbitmq-headless            ClusterIP   None             <none>        4369/TCP,5672/TCP,5671/TCP,25672/TCP,15672/TCP            72m

Other info

MacBook-Pro-3:bin manuchaudhary$ ./istioctl version 
client version: 1.12.0
control plane version: 1.12.0
data plane version: 1.12.0 (16 proxies)

TCP dump inside Istio ingress gateway of a working cluster

15:46:23.647656 IP 122.162.144.32.23435 > 10.0.1.47.5671: Flags [S], seq 3103710824, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 2736535918 ecr 0,sackOK,eol], length 0
15:46:23.647697 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [S.], seq 681655678, ack 3103710825, win 62643, options [mss 8961,sackOK,TS val 3140511074 ecr 2736535918,nop,wscale 7], length 0
15:46:23.717973 IP 122.162.144.32.23435 > 10.0.1.47.5671: Flags [.], ack 1, win 2052, options [nop,nop,TS val 2736535999 ecr 3140511074], length 0
15:46:23.720154 IP 122.162.144.32.23435 > 10.0.1.47.5671: Flags [P.], seq 1:308, ack 1, win 2052, options [nop,nop,TS val 2736536001 ecr 3140511074], length 307
15:46:23.720175 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [.], ack 308, win 487, options [nop,nop,TS val 3140511146 ecr 2736536001], length 0
15:46:23.722885 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [.], seq 1:1429, ack 308, win 487, options [nop,nop,TS val 3140511149 ecr 2736536001], length 1428
15:46:23.722892 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [P.], seq 1429:2857, ack 308, win 487, options [nop,nop,TS val 3140511149 ecr 2736536001], length 1428
15:46:23.722897 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [.], seq 2857:4285, ack 308, win 487, options [nop,nop,TS val 3140511149 ecr 2736536001], length 1428
15:46:23.722899 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [P.], seq 4285:5713, ack 308, win 487, options [nop,nop,TS val 3140511149 ecr 2736536001], length 1428
15:46:23.722921 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [P.], seq 5713:6595, ack 308, win 487, options [nop,nop,TS val 3140511149 ecr 2736536001], length 882
15:46:23.795525 IP 122.162.144.32.23435 > 10.0.1.47.5671: Flags [.], ack 6595, win 1949, options [nop,nop,TS val 2736536075 ecr 3140511149], length 0
15:46:23.803620 IP 122.162.144.32.23435 > 10.0.1.47.5671: Flags [.], ack 6595, win 2048, options [nop,nop,TS val 2736536084 ecr 3140511149], length 0
15:46:23.807625 IP 122.162.144.32.23435 > 10.0.1.47.5671: Flags [P.], seq 308:446, ack 6595, win 2048, options [nop,nop,TS val 2736536086 ecr 3140511149], length 138
15:46:23.807636 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [.], ack 446, win 486, options [nop,nop,TS val 3140511234 ecr 2736536086], length 0
15:46:23.807883 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [P.], seq 6595:6602, ack 446, win 486, options [nop,nop,TS val 3140511234 ecr 2736536086], length 7
15:46:23.808115 IP 10.0.1.47.5671 > 122.162.144.32.23435: Flags [F.], seq 6602, ack 446, win 486, options [nop,nop,TS val 3140511234 ecr 2736536086], length 0
...

TCP dump inside Istio ingress gateway of a NOT WORKING cluster. Notice that F that is sent from gateway.

15:46:43.860865 IP 122.162.144.32.25867 > 10.0.3.179.5671: Flags [S], seq 1911004422, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 1866753148 ecr 0,sackOK,eol], length 0
15:46:43.860889 IP 10.0.3.179.5671 > 122.162.144.32.25867: Flags [S.], seq 3885047813, ack 1911004423, win 62643, options [mss 8961,sackOK,TS val 2941952952 ecr 1866753148,nop,wscale 7], length 0
15:46:44.147954 IP 122.162.144.32.25867 > 10.0.3.179.5671: Flags [.], ack 1, win 2052, options [nop,nop,TS val 1866753437 ecr 2941952952], length 0
15:46:44.148953 IP 122.162.144.32.25867 > 10.0.3.179.5671: Flags [P.], seq 1:308, ack 1, win 2052, options [nop,nop,TS val 1866753440 ecr 2941952952], length 307
15:46:44.148969 IP 10.0.3.179.5671 > 122.162.144.32.25867: Flags [.], ack 308, win 487, options [nop,nop,TS val 2941953240 ecr 1866753440], length 0
15:46:44.149082 IP 10.0.3.179.5671 > 122.162.144.32.25867: Flags [F.], seq 1, ack 308, win 487, options [nop,nop,TS val 2941953240 ecr 1866753440], length 0
15:46:44.455798 IP 122.162.144.32.25867 > 10.0.3.179.5671: Flags [.], ack 2, win 2052, options [nop,nop,TS val 1866753746 ecr 2941953240], length 0
15:46:44.455798 IP 122.162.144.32.25867 > 10.0.3.179.5671: Flags [.], ack 2, win 2052, options [nop,nop,TS val 1866753746 ecr 2941953240], length 0
15:46:44.459798 IP 122.162.144.32.25867 > 10.0.3.179.5671: Flags [F.], seq 308, ack 2, win 2052, options [nop,nop,TS val 1866753750 ecr 2941953240], length 0
15:46:44.459818 IP 10.0.3.179.5671 > 122.162.144.32.25867: Flags [.], ack 309, win 487, options [nop,nop,TS val 2941953551 ecr 1866753750], length 0
...

Upvotes: 0

Views: 692

Answers (1)

manu4543
manu4543

Reputation: 556

So the issue was that there was an Authorization policy in place

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: block-api-clients
  namespace: istio-system
spec:
  action: DENY
  rules:
    - to:
      - operation:
          paths: ["/api/clients/xyz*"]

Not sure about why this was causing istio to close the connection with both downstream and upstream. But removing this fixed the issue.

Upvotes: 1

Related Questions