CODEWITHSUNDEEP
kubernetesapache-kafkaconfluent-platform
Luana Mantovan
Luana Mantovan

Reputation: 35

Confluent for Kubernetes can't access kafka externally

I'm trying to create an external access about confluent-kafka from an AKS cluster. I've been able to connect with control center with an Ingress but i can't create the external access from kafka. I added this to spec of kafka:

listeners:
    external:
      externalAccess:
        type: loadBalancer
        loadBalancer: 
          domain: lb.example.it
          advertisedPort: 39093

and it creates two load-balancer services. Then i added the external IPs on my etc/hosts file:

20.31.10.27    kafka.lb.example.it
20.31.9.167    b0.lb.example.it

But when i ceate a nodejs producer i don't know what to put into broker:

const { Kafka } = require('kafkajs')
const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka.lb.example.it:39093']
})

const producer = kafka.producer()
console.log("produced")


const asyncOperation = async () => {
  console.log("connecting")
await producer.connect()
console.log("connected")
let i = 0

try{
  while(true){
  await producer.send({
    topic: 'topic_prova3',
    messages: [
                {
                  key: JSON.stringify("hello"),
                  value: JSON.stringify({"NUM":i}),
                },
              ]
            })
  await producer.send({
    topic: 'topic_prova3',
    messages: [
                {
                  value: JSON.stringify({"NUMERO":i.toString()}),
                },
              ]
            })
  console.log("sended")
  i++
await new Promise(resolve => setTimeout(resolve, 5000));

}
}
catch(err){
  console.error("error: " + err)
}
await producer.disconnect()
}

asyncOperation();

This is the log of the error:

{"level":"ERROR","timestamp":"2023-01-22T08:43:20.761Z","logger":"kafkajs","message":"[Connection] Connection error: connect ECONNREFUSED 127.0.0.2:9092","broker":"kafka-0.kafka.ckafka.svc.cluster.local:9092","clientId":"my-app","stack":"Error: connect ECONNREFUSED 127.0.0.2:9092\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16)"}

The broker should be kafka.ckafka.svc.cluster.local:9092 , instead of kafka-0.kafka.ckafka.svc.cluster.local:9092 but it does it automatically

Upvotes: 0

Views: 328

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191710

The load balancer appears to be working (it's not throwing unknown host exception), and it returned a cluster local broker address. See above comment that explains why this happens within Kubernetes with Kafka.

You need to modify, or use, the appropriate advertised.listeners port that corresponds to the external LoadBalancer.

from an AKS cluster.

You should use Azure services to create a public DNS route rather than specifically target one IP of your cluster for any given service in /etc/hosts; especially when you'd deploy multiple replicas of the Kafka pod.

Upvotes: 2

Related Questions