CODEWITHSUNDEEP
kubernetesssl-certificatelets-encryptcert-manager
Math
Math

Reputation: 818

Kubernetes Cert-manager issue in Baremetal: Waiting for HTTP-01 challenge propagation: wrong status code '403', expected '200'

Kubernetes Cert-manager on baremetal

I'm trying to configure the certificate for my application, but I'm not able to do that. and the last error that I see is in the Challenge I followed https://cert-manager.io/docs/faq/troubleshooting

Waiting for HTTP-01 challenge propagation: wrong status code '403', expected '200'

The server is a baremetal one.

k describe certificate letsencrypt-something-cert

Name:         letsencrypt-something-cert
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2021-11-29T11:04:21Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .:
          k:{"uid":"2ae6b0b5-46ca-4a56-ab77-3d2fbe2add90"}:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:secretName:
        f:usages:
    Manager:      controller
    Operation:    Update
    Time:         2021-11-29T11:04:21Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:nextPrivateKeySecretName:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2021-11-29T11:04:21Z
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  api-ingress
    UID:                   2ae6b0b5-46ca-4a56-ab77-3d2fbe2add90
  Resource Version:        7041188
  UID:                     a59dedcc-1efe-4778-8ba4-e6a7eee28849
Spec:
  Dns Names:
    something-doe.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  letsencrypt-something-cert
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2021-11-29T11:04:21Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2021-11-29T11:04:21Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  letsencrypt-something-cert-nblzz
Events:                          <none>

k describe ClusterIssuer letsencrypt-staging

Name:         letsencrypt-staging
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2021-11-29T11:21:59Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:acme:
          .:
          f:email:
          f:privateKeySecretRef:
            .:
            f:name:
          f:server:
          f:solvers:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-11-29T11:21:59Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:acme:
          .:
          f:lastRegisteredEmail:
          f:uri:
        f:conditions:
    Manager:         controller
    Operation:       Update
    Subresource:     status
    Time:            2021-11-29T11:22:00Z
  Resource Version:  7036745
  UID:               142f6b6d-b840-4b2e-b50f-278091161b31
Spec:
  Acme:
    Email:            [email protected]
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt-something-cert
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  [email protected]
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/31803711
  Conditions:
    Last Transition Time:  2021-11-29T11:22:00Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   1
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

k describe certificaterequests.cert-manager.io

Name:         letsencrypt-something-cert-fkzs5
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: letsencrypt-something-cert
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: letsencrypt-something-cert-nblzz
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2021-11-29T11:04:21Z
  Generate Name:       letsencrypt-something-cert-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
        f:generateName:
        f:ownerReferences:
          .:
          k:{"uid":"a59dedcc-1efe-4778-8ba4-e6a7eee28849"}:
      f:spec:
        .:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
        f:usages:
    Manager:      controller
    Operation:    Update
    Time:         2021-11-29T11:04:21Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2021-11-29T11:04:21Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  letsencrypt-something-cert
    UID:                   a59dedcc-1efe-4778-8ba4-e6a7eee28849
  Resource Version:        7041199
  UID:                     d885cbc4-babb-457c-bec5-350cbcea5f19
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-6c576bddcf-2qkcp
    authentication.kubernetes.io/pod-uid:
      b1a464a5-6e05-4d89-b95f-fae9f5c4cb35
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-staging
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ213213V2tDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU0wSApaOFpvMWRvQlhXMjJ3eUVtWHhnZXpTQzNCaGFMSzdyOHlSRlR4U2RMWXNzeGk4Z1lia2xJTXJZZE9qV01Ca0lICkpDMDQxelpLZ2RscTN5RWxjL3ZqcVIwdHByRk5oc3lQZWpBOVZHdVVCeFdybGRRZDkyK0dEOFowN3l5QzYwSnAKMlRXTTlFc2ZpVHdmUW1QcDN3TFptZ09TTXVIOU9JajlQYnB6ak9YK1NITXRIYlgwTFkyQ2V2NVlMdVJ1Z3dKUwo2TUgrWnF4VHRwTW9123123KzdUMTBGNDdXdis1S3FPVjl1Vjg2cytDSlF5cXY0QnRWM3lIVGEwb3hDNUYvM1lCcThXCjdrZ3Bxc3d3RTBhR0V3Wnlqbm5rQ3ZpUUg1cFlWM0N3QmNXRThXditpVDRmWXYvVmo5MTlrM0123123pIYzdNenhxSEYKNWd4Q2x0R3AxK1NXOFc2akhSRUNBd0VBQWFBOE1Eb0dDU3FHU0liM0RRRUpEakV0TUNzd0hBWURWUjBSQkJVdwpMFpYSXRabWx1WVc1alpTNWpiMjB3Q3dZRFZSMFBCQVFEQWdXZ01BMEdDU3FHU0liM0RRRUJDd1VBCkE0SUJBUUJIUmFndnhUWWJjLzRJOGJyTVN4MHNUY05MVmFGYzRydkpMY01saFZJbkRrVEl1K2FrTmtYUENNZUMKMDdZUXhZclE1UlZBV1lERGlxYzU4V0x0QXR6TnNieG9JVGVtdENPVFFEU3NXZFh6T3hIckc1aUpEVnEyNW96VQo4WEZWNEFZOFB0VitkUnIyK21LYnh0aUkrMCtIcWxCdFJ1SExYUStvYnpJQ2pVQmw5TWhER2RGdHpzM25ncnEwCjhpeG4xanFFNHlhZTBsRlhtRHFXYU1DODQ4Nm94N1dEZk9Qa2xWNFZhQUJablBEWlREb3FVOFpzUXJPVXZmSm4KOFRKc3Zpd1JNeDZOMjJSN2J0VFRlREhKVjdnbXhmdnB0bFdncEZHc0F5MCtUUDk5YjZBMlVsWUxqbDVXVW45NgpxYTBRTU1aVlVtek14YTYzUGJhdEt5aXhMNHM3Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  UID:      0bac5f49-060b-4c53-baa9-cca209acebbe
  Usages:
    digital signature
    key encipherment
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Conditions:
    Last Transition Time:  2021-11-29T11:04:21Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2021-11-29T11:04:21Z
    Message:               Waiting on certificate issuance from order default/letsencrypt-something-cert-fkzs5-1375084254: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:                    <none>

k get clusterissuers.cert-manager.io

NAME                  READY   AGE
letsencrypt-staging   True    26h
k describe clusterissuers.cert-manager.io
Name:         letsencrypt-staging
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2021-11-29T11:21:59Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:acme:
          .:
          f:email:
          f:privateKeySecretRef:
            .:
            f:name:
          f:server:
          f:solvers:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-11-29T11:21:59Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:acme:
          .:
          f:lastRegisteredEmail:
          f:uri:
        f:conditions:
    Manager:         controller
    Operation:       Update
    Subresource:     status
    Time:            2021-11-29T11:22:00Z
  Resource Version:  7036745
  UID:               142f6b6d-b840-4b2e-b50f-278091161b31
Spec:
  Acme:
    Email:            [email protected]
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt-something-cert
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  [email protected]
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/31803711
  Conditions:
    Last Transition Time:  2021-11-29T11:22:00Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   1
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

k describe orders.acme.cert-manager.io

Name:         letsencrypt-something-cert-fkzs5-1375084254
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: letsencrypt-something-cert
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: letsencrypt-something-cert-nblzz
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2021-11-29T11:04:21Z
  Generation:          1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
        f:ownerReferences:
          .:
          k:{"uid":"d885cbc4-babb-457c-bec5-350cbcea5f19"}:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
    Manager:      controller
    Operation:    Update
    Time:         2021-11-29T11:04:21Z
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:authorizations:
        f:finalizeURL:
        f:state:
        f:url:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2021-11-29T11:04:22Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  letsencrypt-something-cert-fkzs5
    UID:                   d885cbc4-babb-457c-bec5-350cbcea5f19
  Resource Version:        7041110
  UID:                     1bbe18b7-9275-4dc4-872d-bc5f5abf497c
Spec:
  Dns Names:
    something-doe.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-staging
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2dUQ0NBV1232342342tDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU0wSApaOFpvMWRvQlhXMjJ3eUVtWHhnZXpTQzNCaGFMSzdyOHlSRlR4U2RMWXNzeGk4Z1lia2xJTXJZZE9qV01Ca0lICkpDMDQxelpLZ2RscTN5RWxjL3ZqcVIwdHByRk5oc3lQZWpBOVZHdVVCeFdybGRRZDkyK0dEOFowN3l5QzYwSnAKMlRXTTlFc2ZpVHdmUW1QcDN3TFptZ09TTXVIOU9JajlQYnB6ak9YK1NITXRIYlgwNVlMdVJ1Z3dKUwo2TUgrWnF4VHRwTW9kKzdU432423TBGNDdXdis1S3FPVjl1Vjg2cytDSlF5cXY0QnRWM3lIVGEwb3hDNUYvM1lCcThXCjdrZ3Bxc3d3RTBhR0V3Wnlqbm5rQ3ZpUUg1cFlWM0N3QmNXRThXditp234342VDRmWXYvVmo5MTlrM0pIYzdNenhxSEYKNWd4Q2x0R3AxK1NXOFc2akhSRUNBd0VBQWFBOE1Eb0dDU3FHU0liM0RRRUpEakV0TUNzd0hBWURWUjBSQkJVdwpFNElSYjNSMFpYSXRabWx1WV3Q3dZRFZSMFBCQVFEQWdXZ01BMEdDU3FHU0liM0RRRUJDd1VBCkE0SUJBUUJIUmFndnhUWWJjLzRJOGJyTVN4MHNUY05MVmFGYzRydkpMY01saFZJbkRrVEl1K2FrTmtYUENNZUMKMDdZUXhZclE1UlZBV1lERGlxYzU4V0x0QXR6TnNieG9JVGVtdENPVFFEU3NXZFh6T3hIckc1aUpEVnEyNW96VQo4WEZWNEFZOFB0VitkUnIyK21LYnh0aUkrMCtIcWxCdFJ1SExYUStvYnpJQ2pVQmw5TWhER2RGdHpzM25ncnEwCjhpeG4xanFFNHlhsdfZTBsRlhtRHFXYU1DODQ4Nm94N1dEZk9Qa2xWNFZhQUJablBEWlREb3FVOFpzUXJPVXZmSm4KOFRKc3Zpd1JNeDZOMjJSN2J0VFRlREhKVjdnbXhmdnB0bFdncEZHc0F5MCtUUDk5YjZBMlVsWUxqbDVXVW45NgpxYTBRTU1aVlVtek14YTYzUGJhdEt5aXhMNHM3Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
Status:
  Authorizations:
    Challenges:
      Token:        382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/811518411/rvlRdw
      Token:        382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/811518411/nLQlIQ
      Token:        382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/811518411/nccdbA
    Identifier:     something-doe.com
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/811518411
    Wildcard:       false
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/31803711/881473868
  State:            pending
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/31803711/881473868
Events:             <none>

k describe challenges.acme.cert-manager.io

Name:         letsencrypt-something-cert-fkzs5-1375084254-3013399563
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2021-11-29T11:04:22Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizer.acme.cert-manager.io":
        f:ownerReferences:
          .:
          k:{"uid":"1bbe18b7-9275-4dc4-872d-bc5f5abf497c"}:
      f:spec:
        .:
        f:authorizationURL:
        f:dnsName:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:key:
        f:solver:
          .:
          f:http01:
            .:
            f:ingress:
              .:
              f:class:
        f:token:
        f:type:
        f:url:
        f:wildcard:
    Manager:      controller
    Operation:    Update
    Time:         2021-11-29T11:04:22Z
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:presented:
        f:processing:
        f:reason:
        f:state:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2021-11-29T11:04:22Z
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  letsencrypt-something-cert-fkzs5-1375084254
    UID:                   1bbe18b7-9275-4dc4-872d-bc5f5abf497c
  Resource Version:        7041118
  UID:                     4541df79-bb14-4961-bfb1-e068fe2d6c81
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/811518411
  Dns Name:           something-doe.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-staging
  Key:      382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8.Wa-UGUzSApW6EVdbEjJqi-qm7JT76a5T8VhTQOTz2Zo
  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8
  Type:         HTTP-01
  URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/811518411/rvlRdw
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '403', expected '200'
Events:        <none>
  State:       pending

k describe ingress cm-acme-http-solver-9xg4n

Name:             cm-acme-http-solver-9xg4n
Namespace:        default
Address:          139.178.91.167
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  something-doe.com
                     /.well-known/acme-challenge/382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8   cm-acme-http-solver-l55kd:8089 (11.233.0.51:8089)
Annotations:         nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events:              <none>

k describe ingress api-ingress

Name:             api-ingress
Namespace:        default
Address:          139.178.91.167
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  letsencrypt-something-cert terminates something-doe.com
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  something-doe.com
                         /   something-front:5000 (11.233.0.41:5000,11.233.0.42:5000)
  app.something-doe.com
                         /   something-back:3000 (11.233.0.13:3000,11.233.0.14:3000)
Annotations:             cert-manager.io/cluster-issuer: letsencrypt-staging
                         kubernetes.io/ingress.class: nginx
                         nginx.ingress.kubernetes.io/rewrite-target: /
Events:                  <none>

If I call the acme challenge seems to work

curl  http://something-doe.com/.well-known/acme-challenge/322z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8
# 382z5Tloq5AQftHg4q1ldxRPseTDJIZvAR4jQX_L8X8.Wa-UGUzSApW22fdbEjJqi-qm7JT76a5T8VhTQOTz2Zo%

I do not know what else to do/check

Upvotes: 0

Views: 1930

Answers (2)

Clay Risser
Clay Risser

Reputation: 3570

I faced this issue when I was using a NAT rule in pfSense to forward traffic into my cluster. I switched to using the pfSense haproxy and it worked fine after that.

Upvotes: 0

Math
Math

Reputation: 818

Thanks for all the help. the Issue was solved with a workaround https://github.com/jetstack/cert-manager/issues/4003#issuecomment-904420841

basically the cert manager pod was unable to access the service pod. Seems a dns or something blocking the http call, redirecting to https and the cert-manager fails

So to avoid go-out and go-in again

k -n cert-manager edit deployment cert-manager

and add in spec

spec:
  ...
  hostAliases:
    - hostnames:
      - something-do.com
      ip: 138.178.91.111 # server IP

this will add that line to the /etc/hosts so will call directly to your service

Upvotes: 1

Related Questions