CODEWITHSUNDEEP
apache-kafkavirtualboxkafka-consumer-apikafka-producer-api
Florian
Florian

Reputation: 139

Connect to Kafka inside VirtualBox

I'm having troubles connecting from my Host (Windows) to Guest (Linux) where I installed Kafka.

I have set up a VM (with VirtualBox) where I installed Confluent tools. In this VM, I run the command:

confluent start schema-registry

It starts zookeeper, kafka, and schema-registry.

Under this VM, I can run

kafka-console-producer --broker-list localhost:9092 --topic test

and

kafka-console-consumer --bootstrap-server localhost:9092 --topic test

and everything is working fine, I can produce and receive messages.

My goal is however to be able to produce and consume messages from my Host, so I setup this port forwarding rule : port forwarding rule

From my Windows, I was expecting this command to work:

bin\windows\kafka-console-producer.bat --broker-list 127.0.0.1:9092 --topic test

But all I get is this :

ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1529 ms has passed since batch creation plus linger time

I've tried many different stuff but still can't find a solution.... Any ideas?

Upvotes: 6

Views: 5431

Answers (2)

Jeroen
Jeroen

Reputation: 63800

I was trying to set up something similar. Although it's probably better to set up proper virtual networks for Kafka and Zookeeper, I did push on and want to share my personal approach.

I've got things to work with Kafka and Zookeeper running on the VirtualBox Linux guest itself, as well as a docker-based solution running on that Linux guest. Here's the docker-compose.yml I used inside the guest:

version: '3'

services:

  zookeeper:
    image: "wurstmeister/zookeeper"
    ports:
      - "2181:2181"

  kafka:
    image: "wurstmeister/kafka"
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "localhost"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

The crux here is the KAFKA_ADVERTISED_HOST_NAME, which does a dirty little trick that plays well with the default NAT-based networking in VirtualBox (assuming you've patched 2181 and 9092 through as OP did). With this trick any consumer or producer on the VirtualBox Host OS will learn from Kafka that it is located on "localhost", which resolves to the Host, but because the port on the host is patched through to the guest things end up in the right place.

The proper way to do the same in 1.x is to use the Advertised Listeners properties, but the principle remains the same.

For reference, here's my VirtualBox settings:

port 9092 and 2181 forwarded

With those settings, after docker-compose up on the guest OS, I could list/create topics and consume/produce messages from the host OS.

Upvotes: 0

kimutansk
kimutansk

Reputation: 131

It seems to be a host name gap between Guest and Host. What host name does the Linux Guest have?

When producer/consumer accesses the Kafka broker, the Kafka broker returns its host name for data producer or consumer at default settings. So producers/consumers need to resolve broker's host name to IPAddress.

For broker returning an arbitrary host name, use the advertised.listeners settings.

Details are in the "advertised.listeners" configuration docs.

Upvotes: 5

Related Questions