Envoy retry_policy is not working even though the request is routed correctly

Question

I am using following envoy config:

static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: [ "*" ]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: backend_service
                            retry_policy:
                              retry_on: 5xx
                              num_retries: 3
                              per_try_timeout: 10s
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: backend_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: backend_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: service-front
                      port_value: 5000

service-front exposes a worker-pod which returns a 500 response when invoked.

When the envoy service is invoked with curl, the worker responds 500 immediately and I couldn't get the retry to work in any way.

I have tried following approaches,

1. Setting the retry_policy within route as specified in the above yaml
2. Setting the retry_policy within virtual_host
3. Sending x-envoy-retry-on header to the request

None of the above makes the envoy to retry the request. I have enabled debug logs in the envoy but its not printing anything related.