Cannot connect to K8s Kafka cluster from outside of cluster using VIP

Question

I deploy Kafka helm chart https://artifacthub.io/packages/helm/bitnami/kafka/28.3.0 into my k8s cluster with 3 nodes. I also setup an VIP (10.255.251.9) using keepalived. To allow external system to connect to the Kafka cluster, I set the loadBalancerIps to the VIP

externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false
  controller:
    service:
      type: LoadBalancer
      ports:
        external: 9095
      loadBalancerIPs:
        - 10.255.251.9
        - 10.255.251.9
        - 10.255.251.9
  broker:
    service:
      type: LoadBalancer
    ports:
      external: 9095
    loadBalancerIPs:
      - 10.255.251.9
      - 10.255.251.9
      - 10.255.251.9

List of kafka pods

Name                 Ready   Status
kafka-controller-0   1/1     Running  
kafka-controller-1   1/1     Running  
kafka-controller-2   1/1     Running  

List of kafka services

When I try to connect from outside of k8s cluster, I can connect through nodeIP:9095 but NOT VIP:9095.

I log into all the kafka pod the check the server.config.

listeners=CLIENT://:9092,INTERNAL://:9094,EXTERNAL://:9095,CONTROLLER://:9093
advertised.listeners=CLIENT://kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092,INTERNAL://kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9094,EXTERNAL://10.255.251.9:9095
listener.security.protocol.map=CLIENT:SASL_PLAINTEXT,INTERNAL:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

Networkpolicy

Name:         kafka
Namespace:    default
Created on:   2024-11-28 07:47:48 +0000 UTC
Labels:       app.kubernetes.io/instance=kafka
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kafka
              app.kubernetes.io/version=3.7.0
              helm.sh/chart=kafka-28.3.0
              release=kafka
              service=kafka
Annotations:  meta.helm.sh/release-name: kafka
              meta.helm.sh/release-namespace: default
Spec:
  PodSelector:     app.kubernetes.io/instance=kafka,app.kubernetes.io/name=kafka
  Allowing ingress traffic:
    To Port: 9092/TCP
    To Port: 9094/TCP
    To Port: 9093/TCP
    To Port: 9095/TCP
    From: <any> (traffic not restricted by source)
  Allowing egress traffic:
    To Port: <any> (traffic allowed to all ports)
    To: <any> (traffic not restricted by destination)
  Policy Types: Ingress, Egress

Please help!

Here is my full config.

image:
  registry: docker.io
  repository: bitnami/kafka
  tag: 3.7.0-debian-12-r6
  debug: true
containerSecurityContext:
  allowPrivilegeEscalation: false
commonLabels:
  service: kafka
  release: kafka
listeners:
  client:
    containerPort: 9092
    protocol: SASL_PLAINTEXT
    name: CLIENT
    sslClientAuth: none
  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: SASL_PLAINTEXT
    sslClientAuth: none
  interbroker:
    containerPort: 9094
    protocol: SASL_PLAINTEXT
    name: INTERNAL
    sslClientAuth: none
  external:
    containerPort: 9095
    protocol: SASL_PLAINTEXT
    name: EXTERNAL
    sslClientAuth: none
sasl:
  enabledMechanisms: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
  interBrokerMechanism: PLAIN
  controllerMechanism: PLAIN
  interbroker:
    user: inter_broker_user
  controller:
    user: controller_user
  client:
    users:
      - user
  existingSecret: kafka-credentials
kraft:
  enabled: true
allowPlaintextListener: false
controller:
  replicaCount: 3

externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false
  controller:
    service:
      type: LoadBalancer
      ports:
        external: 9095
      loadBalancerIPs:
        - 10.255.251.9
        - 10.255.251.9
        - 10.255.251.9
  broker:
    service:
      type: LoadBalancer
    ports:
      external: 9095
    loadBalancerIPs:
      - 10.255.251.9
      - 10.255.251.9
      - 10.255.251.9
networkPolicy:
  enable: true
  allowExternal: true
  allowExternalEgress: true
serviceAccount:
  create: true
rbac:
  create: true
readinessProbe:
  enabled: true
livenessProbe:
  enabled: true
persistence:
  enabled: true
  size: 10Gi
  annotations:
    helm.sh/resource-policy: keep