CODEWITHSUNDEEP
kubernetesnginx-ingressamazon-eks
porthunt
porthunt

Reputation: 504

Failed request between pods

I have a cluster on EKS with some APIs running on it, this is the yaml file used to deploy them:

apiVersion: v1
kind: Service
metadata:
  name: <api-name>
spec:
  type: ClusterIP
  selector:
    app: <api-name>
  ports:
    - protocol: TCP
      port: 80
      targetPort: <container-port>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: <api-name>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <api-name>
  template:
    metadata:
      labels:
        app: <api-name>
    spec:
      containers:
      - name: <api-name>
        image: <ecr-image-url>
        ports:
        - containerPort: <container-port>
          name: <api-name>
        env:
          - name: ENVIRONMENT
            value: <environment>
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: <api-name>
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: <app-name>.<dns>
    http:
      paths:
      -  backend:
          serviceName: <api-name>
          servicePort: 80

The routing works perfectly ok (Network Load Balancer created by nginx-ingress), but when I try to make a request from one pod to another, I receive a:

[2020-08-14 11:49:42,214] ERROR in app: Exception on /services [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1948, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2242, in preprocess_request
    rv = func()
  File "/app/app/views/__init__.py", line 12, in before_rest_callback
    validate_request(token)
  File "/app/app/utils.py", line 29, in validate_request
    response = httpx.post(url, headers=headers, timeout=60)
  File "/usr/local/lib/python3.8/site-packages/httpx/_api.py", line 269, in post
    return request(
  File "/usr/local/lib/python3.8/site-packages/httpx/_api.py", line 86, in request
    return client.request(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 640, in request
    return self.send(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 670, in send
    response = self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 699, in _send_handling_redirects
    response = self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 736, in _send_handling_auth
    response = self._send_single_request(request, timeout)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 759, in _send_single_request
    (
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_exceptions.py", line 359, in map_exceptions
    raise mapped_exc(message, **kwargs) from None  # type: ignore
httpx._exceptions.ReadError: Server disconnected while attempting read

I can get no connection between pods. The request doesn't reach the other application, running on the same cluster, same node. Nginx Ingress was installed following the official documentation.

Any clues about what might be causing this? I discarded anything related to the deployment (API or gunicorn in this case). It seems to be something related to the cluster and/or nginx-ingress. Tried searching about it and found stuff related to "idle timeout", but that's not applicable to Network Load Balancers.

Upvotes: 1

Views: 670

Answers (1)

porthunt
porthunt

Reputation: 504

Problem solved using http://<app-name> (app-name being the service name). Reference: How to implement Kubernetes POD to POD Communication?

Upvotes: 2

Related Questions