Istio to outside cluster communication issue

Question

In my setup we send all the calls going out of cluster to an Internal Load Balancer in GCP. We do this by creating a egress service and manually adding endpoints to this service. The endpoint to this service is the IP of the Internal load balancer.

[sourabh.w@K9-MAC-035 r19-3]$ k get svc,ep -n egproxy-lle NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/egproxy ClusterIP 10.206.180.135 80/TCP,443/TCP 4d

NAME ENDPOINTS AGE endpoints/egproxy 10.207.132.8:30443,10.207.132.8:30080 4d [sourabh.w@K9-MAC-035 r19-3]$

For all micro-services in my setup, they have to run an "openssl s_Client" command at startup. This command is failing for me.

openssl s_client -servername ae17-api.kohlsecommerce.com -connect ae17-api.kohlsecommerce.com:443 -debug -state

While doing this I make sure ae17-api.kohlsecommerce.com is mapped to egproxy service's IP(10.206.180.135) in /etc/hosts.

Here is the o/p when I run openssl inside pod:

root@product-26-655f4f55b6-g2bpq:/# openssl s_client -servername ae17-api.kohlsecommerce.com -connect ae17-api.kohlsecommerce.com:443 -state -debug CONNECTED(00000003) SSL_connect:before SSL initialization write to 0x556dc50b2860 [0x556dc50c3a20] (212 bytes => 212 (0xD4)) 0000 - 16 03 01 00 cf 01 00 00-cb 03 03 43 59 24 26 31 ...........CY$&1 0010 - 4f 13 80 47 f2 09 25 f7-ec 74 40 57 7c d0 bc c6 O..G..%..t@W|... 0020 - 18 9b a7 a3 3c 38 80 d6-f4 99 62 00 00 38 c0 2c ....<8....b..8., 0030 - c0 30 00 9f cc a9 cc a8-cc aa c0 2b c0 2f 00 9e .0.........+./.. 0040 - c0 24 c0 28 00 00 c0 23-c0 88 00 67 c0 0a c0 14 .$.(.k.#.'.g.... 0050 - 00 39 c0 09 c0 00 00 33-00 9d 00 9c 00 3d 00 3c .9.....3.....=.< 0060 - 00 35 00 2f 00 ff 01 00-66 6a 00 00 00 20 00 1e .5./.....j... .. 0070 - 00 00 1b 61 65 31 37 2d-61 70 69 2e 6b 6f 68 6c ...ae17-api. 0080 - 73 65 63 6f 6d 6d 65 72-63 65 2e 63 6f 6d 00 0b ecommerce.com.. 0090 - 00 04 03 00 01 02 00 8a-00 0a 00 08 00 1d 00 17 ................ 00a0 - 00 19 00 18 00 00 00 66-00 16 00 00 00 17 00 00 .....#.......... 00b0 - 00 0d 00 20 00 00 06 01-06 02 06 03 05 01 05 02 ... ............ 00c0 - 05 03 04 01 04 02 04 03-03 01 03 02 03 03 02 01 ................ 00d0 - 02 02 02 03 .... SSL_connect:SSLv3/TLS write client hello read from 0x556dc50b2860 [0x556dc50ba803] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF)) SSL_connect:error in SSLv3/TLS write client hello

write:errno=104

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 212 bytes

Verification: OK

New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1553126020 Timeout : 7200 (sec) Verify return code: 0 (ok)

Extended master secret: no

I tried creating various set of serviceentries and virtualservices but nothing worked:

With Load Balancer IP:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  generation: 1
  name: egproxy-ext
  namespace: r19-3-mui-qa
spec:
  addresses:
  - 10.207.132.8/32
  endpoints:
  - address: 10.207.132.8
  hosts:
  - istio-ilb.lle-mcommerce.com
  location: MESH_INTERNAL
  ports:
  - name: http
    number: 30080
    protocol: HTTP
  - name: https
    number: 30443
    protocol: HTTPS
  resolution: STATIC

with egproxy serivce FQDN:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: egproxy-headless-service-fqdn-ext
  namespace: r19-3-mui-qa
spec:
  addresses:
  - 10.206.117.116/32
  endpoints:
  - address: 10.207.132.8
  hosts:
  - egproxy.egproxy-lle.svc.cluster.local
  location: MESH_INTERNAL
  ports:
  - name: http
    number: 30080
    protocol: HTTP
  - name: https
    number: 30443
    protocol: HTTPS
  resolution: STATIC

Destination rule for Load balancer:

  apiVersion: networking.istio.io/v1alpha3
  kind: DestinationRule
  metadata:
    name: egress-gateway
    namespace: default
  spec:
    host: istio-ilb.lle-mcommerce.com
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 30443
        tls:
          mode: SIMPLE

Destination rule for egproxy service

Want the communication to work from microservices to ILB via headless egproxy service.

Workarounds like "egress-gateway" is also a viable option but for that also facing problems in putting together correct config to make it work.