blopa
blopa

Reputation: 1

Kafka Kraft docker-compose external access problems

This question is not answered in any thread, it is probably a mistake on Bitnami's part because they told another programmer that the same thing happened to them.If it has happened to someone or has been able to solve it and can share the solution, it would be appreciated.

The question: I have a Kafka kraft in docker-compose inside a Ubuntu server. It works correctly in locally and I can for example, list topics:

~$ docker exec -it blopa-kafka-1 /bin/bash
I have no name!@8b166cb3a647:/$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
__consumer_offsets
boton_topic
dht_topic
gps_topic
pulso_topic

The problem is when trying to connect from another computer (on the same network) with the following code:

KAFKA_BOOTSTRAP_SERVERS = os.getenv('KAFKA_BOOTSTRAP_SERVERS', '192.168.1.211:9092')
app = FastAPI()

def ckeck_conectivity_kafka(kafka_bootstrap_servers):
    kafka_host, kafka_port = kafka_bootstrap_servers.split(':')
    try:

        with socket.create_connection((kafka_host, int(kafka_port)), timeout=10) as sock:

    except Exception as e:
        error_message = f"[ERROR datos_kafka] Not conection {kafka_bootstrap_servers}: {e}"
        logger.error(error_message))

check_conectivity_kafka(KAFKA_BOOTSTRAP_SERVERS)

@asynccontextmanager
async def lifespan(app: FastAPI):
    global producer
    try:
        
        producer = AIOKafkaProducer(
            bootstrap_servers=KAFKA_BOOTSTRAP_SERVERS
        )
        await producer.start()

    except Exception as e:

        await asyncio.sleep(5) 
        yield 
        return 
    yield 
...
    
app.router.lifespan_context = lifespan

This is the trace I get when I run it the code:

INFO:     Started server process [135789]
INFO:     Waiting for application startup.

ERROR:aiokafka:Unable connect to node with id 0: [Errno -2] Name or service not known
KafkaConnectionError: No connection to node with id 0

INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

That is, it does not throw an error, the connection is made but there are problems with the name of service.

This is my .yaml:

services:
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT>
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.211:9092
    volumes:
      - kafka-data:/bitnami/kafka/data
    tty: true
    restart: always
    ports:
      - "9092:9092"
      - "9093:9093"
      - "9094:9094"
volumes:
    kafka-data:
      driver: local

I have been able to connect with telnet to 192.168.1.211:9092 or nc:

nc -zv 192.168.1.211 9092
  192.168.1.211: inverse host lookup failed: Unknown host
  (UNKNOWN) [192.168.1.211] 9092 (?) open

It does not know the service or the host but the port is open but I imagine that some problem in the configuration prevents my code from working.

It's probably not a difficult problem, but solutions such as "you'll need to advertise the external hostname/ip (e.g. 192.168.x.y) of the host as well as/in place of localhost. Requires Docker port forwarding (and firewall / security group changes)" have not worked for me. This is already done and it doesn't work. In fact, if it were not allowed to go through the firewall, the nc or telnet tests would not work.

I add nmap and kcat results:

sudo nmap -p 9092 192.168.1.211 -vv -sV --traceroute

Scanning 192.168.1.211 [1 port] Discovered open port 9092/tcp on 192.168.1.211 HOP RTT ADDRESS 1 53.84 ms 192.168.1.211

echo "Hello, Kafka!" | kcat -P -b 192.168.1.211:9092 -t boton_topic

%3|1727082503.291|FAIL|rdkafka#producer-1| [thrd:8b166cb3a647:9092/0]: 8b166cb3a647:9092/0: Failed to resolve '8b166cb3a647:9092': Name or service not known (after 102ms in state CONNECT) % ERROR: Local: Host resolution failure: 8b166cb3a647:9092/0: Failed to resolve '8b166cb3a647:9092': Name or service not known (after 102ms in state CONNECT) %3|1727082504.353|FAIL|rdkafka#producer-1| [thrd:8b166cb3a647:9092/0]: 8b166cb3a647:9092/0: Failed to resolve '8b166cb3a647:9092': Name or service not known (after 61ms in state CONNECT, 1 identical error(s) suppressed) % ERROR: Local: Host resolution failure: 8b166cb3a647:9092/0: Failed to resolve '8b166cb3a647:9092': Name or service not known (after 61ms in state CONNECT, 1 identical error(s) suppressed)

This is the docker-compose server information:

sudo lsof -i -P -n | grep LISTEN

docker-pr 1624 root 4u IPv4 9165 0t0 TCP *:9092 (LISTEN) docker-pr 1629 root 4u IPv6 9171 0t0 TCP *:9092 (LISTEN)

Upvotes: 0

Views: 48

Answers (0)

Related Questions