CODEWITHSUNDEEP
istioenvoyproxy
svenwltr
svenwltr

Reputation: 18472

How to gzip stats endpoint in Istio?

We are scraping the metrics of many istio-proxy sidecars with Prometheus. As these are many metrics, we would like to compress the payload to save us some bandwidth.

Out of the box the stats endpoint does not seem to be compressed with Istio 1.8.2:

$ kubectl exec -it my-pod-0 -c server -- curl -o /dev/null -vsS --compressed http://127.0.0.1:15090/stats/prometheus
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 15090 (#0)
> GET /stats/prometheus HTTP/1.1
> Host: 127.0.0.1:15090
> User-Agent: curl/7.61.1
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< cache-control: no-cache, max-age=0
< x-content-type-options: nosniff
< date: Fri, 19 Feb 2021 10:42:25 GMT
< server: envoy
< x-envoy-upstream-service-time: 2
< transfer-encoding: chunked
< 
{ [26267 bytes data]
* Connection #0 to host 127.0.0.1 left intact

How do I get the sidecar to compress the stats traffic?

So far I tried adding an EnvoyFilter, but I honestly have no idea about the Envoy internals and I failed to find docs that help me understanding it.

My understanding is that I have to add the compress filter to this:

$ istioctl proxy-config listeners maintenance-0 --port 15090 -o json | gron
json = [];
json[0] = {};
json[0].address = {};
json[0].address.socketAddress = {};
json[0].address.socketAddress.address = "0.0.0.0";
json[0].address.socketAddress.portValue = 15090;
json[0].filterChains = [];
json[0].filterChains[0] = {};
json[0].filterChains[0].filters = [];
json[0].filterChains[0].filters[0] = {};
json[0].filterChains[0].filters[0].name = "envoy.filters.network.http_connection_manager";
json[0].filterChains[0].filters[0].typedConfig = {};
json[0].filterChains[0].filters[0].typedConfig.httpFilters = [];
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0] = {};
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0].name = "envoy.filters.http.router";
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0].typedConfig = {};
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0].typedConfig["@type"] = "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router";
json[0].filterChains[0].filters[0].typedConfig.routeConfig = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts = [];
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0] = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].domains = [];
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].domains[0] = "*";
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].name = "backend";
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes = [];
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0] = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].match = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].match.prefix = "/stats/prometheus";
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].route = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].route.cluster = "prometheus_stats";
json[0].filterChains[0].filters[0].typedConfig.statPrefix = "stats";
json[0].filterChains[0].filters[0].typedConfig["@type"] = "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager";

So far I tried creating the filter a few times and this is my latest try:

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gzip
spec:
  workloadSelector:
    labels:
      app: my-pod
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
              subFilter:
                name: envoy.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.compressor
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
            compressor_library:
              name: text_optimized
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
            remove_accept_encoding_header: true

I do not really know what to put into the .spec.configPatches.match section. The patch and applyTo section probably are wrong too.

Upvotes: 2

Views: 1475

Answers (1)

svenwltr
svenwltr

Reputation: 18472

With help in an Istio issue, we made it work. I am copying my original response from: https://github.com/istio/istio/issues/30987#issuecomment-822517456

I got a working example and our network usage went down from ~20MBytes/s to ~30KBytes/s (yes, from Mega to Kilo 🔥). First I thought there was any error, but the data was complete and I did a short check with my CLI:

$ kubectl exec elasticsearch-0 -c istio-proxy -- timeout 1 curl -Ss --fail --compressed -w '%{size_download}' -i http://localhost:14090/stats/prometheus | tail -n 1
7763

$ kubectl exec elasticsearch-0 -c istio-proxy -- timeout 1 curl -Ss --fail -w '%{size_download}' -i http://localhost:14090/stats/prometheus | tail -n 1          
330315

It is only 2.35% of its original size and both have the same mount of lines!

Here is the custom bootstrap, that need to be added to each pod with the sidecar.istio.io/bootstrapOverride: "istio-custom-bootstrap-config" annotation.

apiVersion: v1
kind: ConfigMap

metadata:
  annotations:
  name: istio-custom-bootstrap-config
  namespace: default

data:
  custom_bootstrap.json: |-
    {
        "staticResources": {
            "listeners": [
                {
                    "address": {
                        "socketAddress": {
                            "address": "0.0.0.0",
                            "portValue": 14090
                        }
                    },
                    "filterChains": [
                        {
                            "filters": [
                                {
                                    "name": "envoy.filters.network.http_connection_manager",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                                        "httpFilters": [
                                            {
                                                "name": "envoy.filters.http.compressor",
                                                "typed_config": {
                                                    "@type": "type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor",
                                                    "compressor_library": {
                                                        "name": "text_optimized",
                                                        "typed_config": {
                                                            "@type": "type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip"
                                                        }
                                                    },
                                                    "remove_accept_encoding_header": true
                                                }
                                            },
                                            {
                                                "name": "envoy.filters.http.router",
                                                "typedConfig": {
                                                    "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                                                }
                                            }
                                        ],
                                        "routeConfig": {
                                            "virtualHosts": [
                                                {
                                                    "domains": [
                                                        "*"
                                                    ],
                                                    "name": "backend",
                                                    "routes": [
                                                        {
                                                            "match": {
                                                                "prefix": "/stats/prometheus"
                                                            },
                                                            "route": {
                                                                "cluster": "prometheus_stats"
                                                            }
                                                        }
                                                    ]
                                                }
                                            ]
                                        },
                                        "statPrefix": "stats"
                                    }
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    }

I had to change the port, because it is easy to maintain when there gets anything added to staticResources.listeners in future updates.

Upvotes: 2

Related Questions