CODEWITHSUNDEEP
amazon-web-serviceskubernetesamazon-ekskubernetes-networkpolicy
Skynet
Skynet

Reputation: 578

Egress NetworkPolicy in AWS EKS not allowing traffic to service's pods

I have an application that is deployed using Helm inside an AWS EKS cluster with VPC CNI enabled for Network Policies. That has worked so far using only ingress allowlist policies, but unfortunately a new requirement is that also egress has to be allowlisted.

Despite adding an egress rule to the backend's network policy, it is unable to contact the Redis service (and eventually, wait-for-it fails as it couldn't contact Redis). DNS resolution is not the culprit, that one got fixed after I added an egress policy to all pods allowing egress to TCP/UDP 53 in the kube-system namespace (and thus, I also see that egress allowing works in principle).

Any idea what I am missing? Is there maybe a way to show the pods allowed by a selector rule combination in kubectl describe networkpolicy similar to how kubectl describe service shows the currently active endpoints?

The AWS VPC CNI logs of the host where the backend pod's init container show a DENY for the traffic (as expected):

{"level":"info","ts":"2024-11-13T20:13:43.120Z","logger":"ebpf-client","msg":"Flow Info:  ","Src IP":"198.18.0.177","Src Port":37827,"Dest IP":"192.168.28.141","Dest Port":6379,"Proto":"TCP","Verdict":"DENY"}

The source IP is the one of the backend pod:

$ kubectl get pod -o custom-columns=NAME:metadata.name,IP:status.podIP,HOST:status.hostIP|grep -e '198.18.0.177'
backend-branch-develop-84b68c4695-w9r49                           198.18.0.177    198.18.3.63

But curiously, the destination IP seems to be wrong - it tries to filter for the service's IP instead of whatever IP the individual Redis pod has:

$ kubectl get pod -o custom-columns=NAME:metadata.name,IP:status.podIP,HOST:status.hostIP|grep -e 'redis-branch-develop'
redis-branch-develop-6c96759874-rzbfk                             198.18.7.113    198.18.3.100
$ kubectl get service -o custom-columns=NAME:metadata.name,IP:spec.clusterIP|grep -e '192.168.28.141'
redis-branch-develop                                         192.168.28.141

$ kubectl get namespace --selector=kubernetes.io/metadata.name=mynamespace
NAME                          STATUS   AGE
mynamespace   Active   460d
$ kubectl get pod --selector=app=redis-branch-develop,release-name=branch-develop,software-name=myapp
NAME                                    READY   STATUS    RESTARTS   AGE
redis-branch-develop-6c96759874-rzbfk   1/1     Running   0          3h13m
$ kubectl get pod --selector=app=backend-branch-develop,release-name=branch-develop,software-name=myapp
NAME                                      READY   STATUS     RESTARTS        AGE
backend-branch-develop-84b68c4695-w9r49   0/1     Init:0/3   5 (5m12s ago)   36m
$ kubectl describe service redis-branch-develop
Name:              redis-branch-develop
Namespace:         mynamespace
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: branch-develop
                   meta.helm.sh/release-namespace: mynamespace
Selector:          app=redis-branch-develop
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                192.168.28.141
IPs:               192.168.28.141
Port:              redis  6379/TCP
TargetPort:        6379/TCP
Endpoints:         198.18.7.113:6379
Session Affinity:  None
Events:            <none>
$ kubectl describe networkpolicy myapp-netpol-allow-backend-branch-develop myapp-netpol-allow-redis-branch-develop myapp-netpol-allowdns-branch-develop myapp-netpol-denyall-branch-develop
Name:         myapp-netpol-allow-backend-branch-develop
Namespace:    mynamespace
Created on:   2024-11-13 13:11:51 +0100 CET
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: branch-develop
              meta.helm.sh/release-namespace: mynamespace
Spec:
  PodSelector:     app=backend-branch-develop,release-name=branch-develop,software-name=myapp
  Allowing ingress traffic:
    To Port: 1337/TCP
    From:
      PodSelector: app.kubernetes.io/component=controller,app.kubernetes.io/name=ingress-nginx
  Allowing egress traffic:
    To Port: 6379/TCP
    To:
      NamespaceSelector: kubernetes.io/metadata.name=mynamespace
      PodSelector: app=redis-branch-develop,release-name=branch-develop,software-name=myapp
  Policy Types: Ingress, Egress


Name:         myapp-netpol-allow-redis-branch-develop
Namespace:    mynamespace
Created on:   2024-11-13 13:11:51 +0100 CET
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: branch-develop
              meta.helm.sh/release-namespace: mynamespace
Spec:
  PodSelector:     app=redis-branch-develop,release-name=branch-develop,software-name=myapp
  Allowing ingress traffic:
    To Port: 6379/TCP
    From:
      NamespaceSelector: kubernetes.io/metadata.name=mynamespace
      PodSelector: app=backend-branch-develop,release-name=branch-develop,software-name=myapp
  Not affecting egress traffic
  Policy Types: Ingress

Name:         myapp-netpol-allowdns-branch-develop
Namespace:    mynamespace
Created on:   2024-11-13 20:12:15 +0100 CET
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: branch-develop
              meta.helm.sh/release-namespace: mynamespace
Spec:
  PodSelector:     release-name=branch-develop,software-name=myapp
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: 53/TCP
    To Port: 53/UDP
    To:
      NamespaceSelector: kubernetes.io/metadata.name=kube-system
      PodSelector: k8s-app=kube-dns
  Policy Types: Egress


Name:         myapp-netpol-denyall-branch-develop
Namespace:    mynamespace
Created on:   2024-11-13 13:11:51 +0100 CET
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: branch-develop
              meta.helm.sh/release-namespace: mynamespace
Spec:
  PodSelector:     release-name=branch-develop,software-name=myapp
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Ingress, Egress

Upvotes: 0

Views: 83

Answers (1)

Fedi Bounouh
Fedi Bounouh

Reputation: 1356

You are trying to communicate through the clusterIP service, not the the redis POD, that's why the destination policy seen by network policy is the service POD while it's configured to allow communication for the POD instead. so to solve the issue you have to update the egress policy to include the clusterIP also

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: myapp-netpol-allow-backend-branch-develop
  namespace: mynamespace
spec:
  podSelector:
    matchLabels:
      app: backend-branch-develop
      release-name: branch-develop
      software-name: myapp
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: redis-branch-develop
          release-name: branch-develop
          software-name: myapp
      namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: mynamespace
    ports:
    - protocol: TCP
      port: 6379
  - to:
    - ipBlock:
        cidr: 192.168.28.141/32
    ports:
    - protocol: TCP
      port: 6379

If you only have an egress rule allowing connections to Redis pods (via PodSelector), traffic routed through the service will be blocked, because the egress rule does not match the service’s ClusterIP, the previous configuration will allow now both, egress to redis pod directly, and egress to the clusterIP as well.

Upvotes: 0

Related Questions