Error "upstream connect error or disconnect/reset before headers" when trying to proxy some static resources

Question

I'm new to envoy. I'm trying to proxy some swagger resources requests, through the same port I'm using for grpc, using envoy, but so far no luck. I keep getting the error below, when calling the swagger resourced. Although according to the envoy logs, the requests seem to reach the BE swagger server. The GRPC APIs are working fine though. Also the swagger resources are available if I access directly the back-end server.

upstream connect error or disconnect/reset before headers

Any idea what I'm doing wrong? Thanks

My envoy conf file looks like this:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: mix-api-listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 51051 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/swagger-ui"}
                route: { cluster: swagger-backend-services, timeout: { seconds: 1500 } }
              - match: { prefix: "/v2"}
                route: { cluster: swagger-backend-services, timeout: { seconds: 1500 } }
              - match: { prefix: "/", grpc: {} }
                route: { cluster: grpc-backend-services, timeout: { seconds: 120 } }
          http_filters:
          - name: envoy.grpc_json_transcoder
            config:
              proto_descriptor: "/data/mix-api.pb"
              services: ["mix.api.MixAPI"]
              print_options:
                add_whitespace: true
                always_print_primitive_fields: true
                always_print_enums_as_ints: false
                preserve_proto_field_names: false
          - name: envoy.router

  clusters:
  - name: grpc-backend-services
    connect_timeout: 1.25s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: ${GRPC_HOST}
        port_value: 9090
  - name: swagger-backend-services
    connect_timeout: 15s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: ${GRPC_HOST}
        port_value: 8080

Envoy logs:

mix-api-envoy_1  | [2020-11-12 20:13:52.001][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:52.002][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:13:52.002][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:52.015][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:52.015][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:13:52.016][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:57.005][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:57.005][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:13:57.005][000009][debug][main] [source/server/server.cc:143] flushing stats
mix-api-envoy_1  | [2020-11-12 20:13:57.006][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:57.017][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:13:57.018][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:13:57.018][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:02.008][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:02.009][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:02.009][000009][debug][main] [source/server/server.cc:143] flushing stats
mix-api-envoy_1  | [2020-11-12 20:14:02.013][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:02.020][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:02.020][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:02.022][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:04.396][000015][debug][main] [source/server/connection_handler_impl.cc:257] [C0] new connection
mix-api-envoy_1  | [2020-11-12 20:14:04.396][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C0] socket event: 2
mix-api-envoy_1  | [2020-11-12 20:14:04.396][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C0] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C0] socket event: 3
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C0] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][connection] [source/common/network/connection_impl.cc:477] [C0] read ready
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C0] read returns: 533
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C0] read returns: -1
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][connection] [source/common/network/raw_buffer_socket.cc:29] [C0] read error: 11
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][http] [source/common/http/http1/codec_impl.cc:348] [C0] parsing 533 bytes
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][http] [source/common/http/http1/codec_impl.cc:446] [C0] message begin
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][debug][http] [source/common/http/conn_manager_impl.cc:200] [C0] new stream
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Host value=localhost:51051
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Connection value=keep-alive
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Pragma value=no-cache
mix-api-envoy_1  | [2020-11-12 20:14:04.397][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Cache-Control value=no-cache
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Upgrade-Insecure-Requests value=1
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=User-Agent value=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Accept value=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Accept-Encoding value=gzip, deflate, br
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Accept-Language value=en-US,en;q=0.9,fr;q=0.8,ro;q=0.7
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:412] [C0] headers complete
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:316] [C0] completed header: key=Cookie value=_ga=GA1.1.645352564.1586804903
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:433] [C0] message complete
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][http] [source/common/http/conn_manager_impl.cc:537] [C0][S17149217713388930625] request headers complete (end_stream=true):
mix-api-envoy_1  | ':authority', 'localhost:51051'
mix-api-envoy_1  | ':path', '/swagger-ui/index.html'
mix-api-envoy_1  | ':method', 'GET'
mix-api-envoy_1  | 'connection', 'keep-alive'
mix-api-envoy_1  | 'pragma', 'no-cache'
mix-api-envoy_1  | 'cache-control', 'no-cache'
mix-api-envoy_1  | 'upgrade-insecure-requests', '1'
mix-api-envoy_1  | 'user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
mix-api-envoy_1  | 'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'
mix-api-envoy_1  | 'accept-encoding', 'gzip, deflate, br'
mix-api-envoy_1  | 'accept-language', 'en-US,en;q=0.9,fr;q=0.8,ro;q=0.7'
mix-api-envoy_1  | 'cookie', '_ga=GA1.1.645352564.1586804903'
mix-api-envoy_1  | 
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][http] [source/common/http/conn_manager_impl.cc:953] [C0][S17149217713388930625] request end stream
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/conn_manager_impl.cc:772] [C0][S17149217713388930625] decode headers called: filter=0x3d3af00 status=0
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][router] [source/common/router/router.cc:270] [C0][S17149217713388930625] cluster 'swagger-backend-services' match for URL '/swagger-ui/index.html'
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][router] [source/common/router/router.cc:328] [C0][S17149217713388930625] router decoding headers:
mix-api-envoy_1  | ':authority', 'localhost:51051'
mix-api-envoy_1  | ':path', '/swagger-ui/index.html'
mix-api-envoy_1  | ':method', 'GET'
mix-api-envoy_1  | ':scheme', 'http'
mix-api-envoy_1  | 'pragma', 'no-cache'
mix-api-envoy_1  | 'cache-control', 'no-cache'
mix-api-envoy_1  | 'upgrade-insecure-requests', '1'
mix-api-envoy_1  | 'user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
mix-api-envoy_1  | 'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'
mix-api-envoy_1  | 'accept-encoding', 'gzip, deflate, br'
mix-api-envoy_1  | 'accept-language', 'en-US,en;q=0.9,fr;q=0.8,ro;q=0.7'
mix-api-envoy_1  | 'cookie', '_ga=GA1.1.645352564.1586804903'
mix-api-envoy_1  | 'x-forwarded-proto', 'http'
mix-api-envoy_1  | 'x-request-id', '0159522f-2f2a-432f-bb31-ab186a77a0c2'
mix-api-envoy_1  | 'x-envoy-expected-rq-timeout-ms', '1500000'
mix-api-envoy_1  | 
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][client] [source/common/http/codec_client.cc:26] [C1] connecting
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][connection] [source/common/network/connection_impl.cc:634] [C1] connecting to 172.19.0.3:8080
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][connection] [source/common/network/connection_impl.cc:643] [C1] connection in progress
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][http2] [source/common/http/http2/codec_impl.cc:721] [C1] setting stream-level initial window size to 268435456
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][http2] [source/common/http/http2/codec_impl.cc:743] [C1] updating connection-level initial window size to 268435456
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/conn_manager_impl.cc:772] [C0][S17149217713388930625] decode headers called: filter=0x3d3af50 status=1
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http] [source/common/http/http1/codec_impl.cc:369] [C0] parsed 533 bytes
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:282] [C0] readDisable: enabled=true disable=true
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C1] socket event: 2
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C1] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][connection] [source/common/network/connection_impl.cc:516] [C1] connected
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][client] [source/common/http/codec_client.cc:64] [C1] connected
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][pool] [source/common/http/http2/conn_pool.cc:83] [C1] creating stream
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][debug][router] [source/common/router/router.cc:1023] [C0][S17149217713388930625] pool ready
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C1] send data: bytes=24
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:376] [C1] writing 24 bytes, end_stream false
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C1] send data: bytes=21
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:376] [C1] writing 21 bytes, end_stream false
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C1] sent frame type=4
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C1] send data: bytes=13
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:376] [C1] writing 13 bytes, end_stream false
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C1] sent frame type=8
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C1] send data: bytes=412
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:376] [C1] writing 412 bytes, end_stream false
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C1] sent frame type=1
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C1] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C1] write returns: 470
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C0] socket event: 2
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C0] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C1] socket event: 2
mix-api-envoy_1  | [2020-11-12 20:14:04.398][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C1] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C1] socket event: 3
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C1] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:477] [C1] read ready
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C1] read returns: 0
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][connection] [source/common/network/connection_impl.cc:501] [C1] remote close
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][connection] [source/common/network/connection_impl.cc:183] [C1] closing socket: 0
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][client] [source/common/http/codec_client.cc:82] [C1] disconnect. resetting 1 pending requests
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][client] [source/common/http/codec_client.cc:105] [C1] request reset
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][pool] [source/common/http/http2/conn_pool.cc:222] [C1] destroying stream: 0 remaining
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][router] [source/common/router/router.cc:481] [C0][S17149217713388930625] upstream reset
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][http] [source/common/http/conn_manager_impl.cc:1083] [C0][S17149217713388930625] encode headers called: filter=0x3c80580 status=0
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][http] [source/common/http/conn_manager_impl.cc:1188] [C0][S17149217713388930625] encoding headers via codec (end_stream=false):
mix-api-envoy_1  | ':status', '503'
mix-api-envoy_1  | 'content-length', '57'
mix-api-envoy_1  | 'content-type', 'text/plain'
mix-api-envoy_1  | 'date', 'Thu, 12 Nov 2020 20:14:03 GMT'
mix-api-envoy_1  | 'server', 'envoy'
mix-api-envoy_1  | 
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:376] [C0] writing 134 bytes, end_stream false
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][http] [source/common/http/conn_manager_impl.cc:1291] [C0][S17149217713388930625] encode data called: filter=0x3c80580 status=0
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][http] [source/common/http/conn_manager_impl.cc:1304] [C0][S17149217713388930625] encoding data via codec (size=57 end_stream=true)
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:376] [C0] writing 57 bytes, end_stream false
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:282] [C0] readDisable: enabled=false disable=false
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][pool] [source/common/http/http2/conn_pool.cc:136] [C1] client disconnected
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][debug][pool] [source/common/http/http2/conn_pool.cc:160] [C1] destroying primary client
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=3)
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=3)
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:439] [C0] socket event: 2
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/connection_impl.cc:507] [C0] write ready
mix-api-envoy_1  | [2020-11-12 20:14:04.402][000015][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C0] write returns: 191
mix-api-envoy_1  | [2020-11-12 20:14:07.011][000009][debug][main] [source/server/server.cc:143] flushing stats
mix-api-envoy_1  | [2020-11-12 20:14:07.014][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:07.015][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:07.016][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:07.024][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:07.024][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:07.024][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
^Amix-api-envoy_1  | [2020-11-12 20:14:12.013][000009][debug][main] [source/server/server.cc:143] flushing stats
mix-api-envoy_1  | [2020-11-12 20:14:12.017][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:12.017][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:12.018][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:12.025][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:12.026][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:12.026][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:17.015][000009][debug][main] [source/server/server.cc:143] flushing stats
mix-api-envoy_1  | [2020-11-12 20:14:17.019][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:17.019][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:17.020][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:17.027][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:80] starting async DNS resolution for mix-api
mix-api-envoy_1  | [2020-11-12 20:14:17.027][000009][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
mix-api-envoy_1  | [2020-11-12 20:14:17.027][000009][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:88] async DNS resolution complete for mix-api

Answer 1

Increase the cluster timeouts of the clusters or Remove http2_protocol_options: {} field from the clusters and try again. Maybe it will fix this.