CODEWITHSUNDEEP
kubeflowkubeflow-kserve
TaeUk Noh
TaeUk Noh

Reputation: 146

I run the kserve example for sklearn-iris and got `302 Found`

serving exam model

  1. create namespace

    $ kubectl create namespace kserve-test

  2. create InferenceService

    $ kubectl apply -n kserve-test -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

    check

    $ kubectl get inferenceservices sklearn-iris -n kserve-test

    NAME           URL                                           READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION                    AGE
sklearn-iris   http://sklearn-iris.kserve-test.example.com   True           100                              sklearn-iris-predictor-default-00001   5h11m

  3. check SERVICE_HOSTNAME, INGRESS_PORT, INGRESS_HOST

    • SERVICE_HOSTNAME

      $ SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
$ echo $SERVICE_HOSTNAME

      sklearn-iris.kserve-test.example.com

    • INGRESS_PORT

      $ INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
$ echo $INGRESS_PORT

      31018

    • INGRESS_HOST

      $ INGRESS_HOST=192.168.219.100

      192.168.219.100 : Internal IP of my device

  4. create input

    $ cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

  5. send request

    $ curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d @./iris-input.json

    *   Trying 192.168.219.100:31018...
* Connected to 192.168.219.100 (192.168.219.100) port 31018 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kserve-test.example.com
> User-Agent: curl/7.71.1
> Accept: */*
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MDMxOHxFd3dBRUhZek16WktlRlZJZFc1alowVlROVTA9fFynQ-3082qPF_-qUwnYllySrEQPAKGqpBuF-Pu9gcnx
< date: Wed, 21 Dec 2022 02:38:38 GMT
< x-envoy-upstream-service-time: 6
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 192.168.219.100 left intact

    Got 302 Found.

    As far as I know, this result is because I was asked for authentication from Dex.

what i did to solve this

  1. Authentication

    I tried to get the authservice_session token by following the method here: kserve:github

    1. From CLI

      $ CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')

      CLUSTER_IP: 10.103.239.220

      $ curl -v http://${CLUSTER_IP}

      *   Trying 10.103.239.220:80...
* Connected to 10.103.239.220 (10.103.239.220) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.103.239.220
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM
< date: Wed, 21 Dec 2022 03:26:01 GMT
< content-length: 269
< x-envoy-upstream-service-time: 4
< server: istio-envoy
< 
<a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Flogin%2Foidc&amp;response_type=code&amp;scope=profile+email+groups+openid&amp;state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM">Found</a>.

* Connection #0 to host 10.103.239.220 left intact

      I stuck at this stage. I think it's wrong to see 302 Found.

    2. From the browser

      Copy the token content from the cookie authservice_session

      $ SESSION =MTY3MTUyODQ2M3xOd3dBTkRkRVExbEdVa0kzVFRJMFMwOU9VRE5hV2pSS1VGVkNSRVJVUlRKVlVVOUlTa2hDVWpOU1RUZFRVRkJGVTFGV1N6UktXVkU9fCoQdbMu_diLBJAKLZSmF4qoqQTlINKq7A63hy-QNQcR

      $ curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}"  http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d ./iris-input.json

      got 404 Not Found

      curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d @./iris-input.json

      got 404 Not Found

  2. added Envoy filter for bypass Dex

    $ vi envoyfilter.yaml

    apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: sklearn-iris-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          name: sklearn-iris.kserve-test.example.com:31018
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: true

    $ kubectl apply -f envoyfilter.yaml

    Note: spec.configPatches.match.routeConfiguration.vhost.name : sklearn-iris.kserve-test.example.com:31018

    It's not working.

    Got still 302 Found

  3. External Authorization

    reference: https://github.com/kubeflow/kubeflow/issues/4549#issuecomment-932259673

    add AuthorizationPolicy

    $ vi authorizationpolicy.yaml

    apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: dex-auth
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    name: dex-auth-provider
  rules:
  # The rules specify when to trigger the external authorizer.
    - to:
      - operation:
          notPaths: ["/v1*"]

    $ kubectl apply -f authorizationpolicy.yaml

    Note: rules.to.operation: notPaths: ["/v1*"]

    and delete envoyfilters named authn-filter that originally existed

    $ kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter

    next, restart deployment/istiod

    $ kubectl rollout restart deployment/istiod -n istio-system

    It's not working.

    Got still 302 Found if don't delete envoyfilters named authn-filter that originally existed, or block the connection if I delete authn-filter .

What I need help:

env:

Upvotes: 1

Views: 1302

Answers (1)

Neil Wang
Neil Wang

Reputation: 11

I met the same issue, please try to specify the session by --cookie instead of apply it in the headers.

curl --cookie "authservice_session=${SESSION}" -v -H "Host: ${SERVICE_HOST}" "http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict" -d @./iris-input.json

Upvotes: 1

Related Questions