Anthos Service Mesh and Istio Ingress Gateway on GKE - Is it the right solution for Multi-Cluster services?

Question

I have deployed a multi-cluster application on GKE using Anthos Service Mesh, but I have some doubt about the available options, their pro and cons. Both I have tested and are working, but need help to go further. We need some additional control of traffic between different cluster services, as we would like to deploy some service only in one cluster, because, for example, they are closer to the main DB that is deployed in one of the two regions. Example of the ideal configuration (simplified):

• Frontend service deployed in both clusters
• Api service deployed only in one cluster
• traffic from external load balancer routed to the nearest cluster Frontend service
• Both Frontends connect to the single Api instance

This is what I tried:

• First solution (easy): deploy a MultiClusterService and a MultiClusterIngress to expose the services deployed in both GKE clusters to obtain a global load balancing with auto-route of traffic to the nearest cluster. Api service must be deployed in both clusters. This solution follow this sample

• Second solution (harder): use istio-ingressgateway deployed in both clusters to expose services using VirtualService and DestinationRule Istio configurations, then expose the gateways behind global MultiClusterService and MultiClusterIngress. This kind of configuration comes from this sample.

The first solution, doesn't allow to manage fine-grain inter-cluster service traffic based on source, http headers etc., my solution was to deploy all services in both clusters and don't know how to manage routing from a service in one cluster to one in the other cluster (Frontend -> Api) (any tips about this?)

The second solution allow inter-services routing (using DestinationRule) but seems that traffic load balancing with auto-route to nearest cluster is missing, only round-robin, least connect and other options are available (see Istio LB options). The LocalityLBSetting seems to work but is really hard and boilerplate to configure with two regions and 6 zones and, again, the automatic route to the nearest cluster is missing. The source cluster label option (Istio Partitioning Service) isn't accepted by GKE because topology.istio.io/cluster is not valid, i don't know why.

Before spending a lot of time to find what's working or not, my questions are:

• Are "apiVersion: networking.istio.io/v1alpha3" configuration files used for Istio gateways, virtual services etc. still valid or are going to be deprecated from GKE in the future (see Istio on GKE deprecated after Sep. 2022)
• Are there other options not using basic Istio configurations to manage inter-service / inter-cluster communications using only global MultiClusterIngress as external LB and global routing?

I've read about Traffic Director that seems to be a new method to manage traffic about services, but can't understand how does it fit with Anthos, MCS, MCI and my configuration.

Any help will be appreciated. Thank you

Addendum: this is my current DestinationRule configuration

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: web-front-v2-multi-cluster
  namespace: staging
spec:
  host: my-front-v2-multi-cluster.staging.cluster.local
  trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
        localityLbSetting:
          enabled: true
          failoverPriority:
          - "topology.kubernetes.io/region"
          - "topology.kubernetes.io/zone"
          - "topology.istio.io/cluster"
      outlierDetection:
        consecutive5xxErrors: 5
        interval: 15s
        baseEjectionTime: 30s
        maxEjectionPercent: 100
  subsets:
  - name: europe
    labels:
      location: cluster-1
      topology.istio.io/cluster: cn-my-project-europe-west2-cluster-1
  - name: america
    labels:
      location: cluster-2
      topology.istio.io/cluster: cn-my-project-northamerica-northeast1-cluster-2

Answer 1

ASM does support multi-cluster failover with locality-aware routing using either the failover or failoverPriority setting:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: whereami-app-1-dr
  namespace: app-1
spec:
  host: whereami.app-1.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    loadBalancer:
      simple: ROUND_ROBIN
      localityLbSetting:
        enabled: true
        # If more than 2 targets you can specify explicit failover options
        # failover:
        #  - from: us-west1
        #    to: us-west3
        # or you can specify label priority for failover https://istio.io/latest/docs/reference/config/networking/destination-rule/#LocalityLoadBalancerSetting
        failoverPriority:
        - "topology.kubernetes.io/region"
        - "topology.kubernetes.io/zone"
        - "topology.istio.io/cluster"
    outlierDetection:        # Required for locality aware routing
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 1m
  subsets:
  - name: primary
    labels:
      # cluster named gke-oregon in us-west1 region of project my-vpc
      topology.istio.io/cluster: cn-my-vpc-us-west1-gke-oregon
  - name: secondary
    labels:
      # cluster named gke-slc in us-west3 region of project my-vpc
      topology.istio.io/cluster: cn-my-vpc-us-west3-gke-slc

Also the special label to match the GKE clusterID is topology.istio.io/cluster: cn-<Project Name>-<Region/Zone>-<Cluster Name>, which can be used to create cluster specific subsets or virtual service matches on sourceLabels