CODEWITHSUNDEEP
javaapache-kafkakafka-producer-apiwindows-subsystem-for-linux
Viriato
Viriato

Reputation: 3011

Unable to produce to Kafka topic that is running on WSL 2 from Windows

I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij on Windows and create a simple Java Producer it does not seem to be able to connect to the broker

Versions & Hostname

    Java version: 1.8
    Kafka Version: 2.6
    hostname (from Ubuntu): KDAAPPDEV04
    hostname (from Powershell): KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
    netstat from CMD:
        TCP    [::1]:9092             [::]:0                 LISTENING

server.properties I found this settings on another SO answer but these did not work for me.

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

then tried (and restarted zookeeper and kafka)

advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

Producer

I run this producer with three different values: hostname, localhost and 127.0.0.1 but it never connects to the broker

    public class ProducerDemo{

    private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);

    public static void main(String[] args) throws UnknownHostException{

        System.out.println(InetAddress.getLocalHost().getHostName());
        System.out.println(InetAddress.getLocalHost().getCanonicalHostName());

        String bootstrapServers = "127.0.0.1:9092";
//        String bootstrapServers = "localhost:9092";
//        String bootstrapServers = "KDAAPPDEV04:9092";

        //create Producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);

        //create a producer record
        ProducerRecord<String,String> record = new ProducerRecord<String, String>("first-topic","hola mundo");

        //send data
        producer.send(record);

        //flush + close
        producer.flush();
        producer.close();
    }
}

Error

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) could not be established. Broker may not be available.

Upvotes: 14

Views: 11303

Answers (10)

Hyperdingo
Hyperdingo

Reputation: 311

I use docker inside my WSL2 linux instance.

I use this base docker compose to run kafka: https://github.com/conduktor/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml

I had to modify the file and replace: ${DOCKER_HOST_IP:-127.0.0.1} on 2 places with the private ip. In linux you can find that ip by writing: ip addr | grep "eth0", e.g. 172.x.y.z

After that I started the docker instance with: docker compose -f zk-single-kafka-single.yml up -d

Then I could connect to kafka with above aquired ip, e.g. 172.x.y.z:9092

Upvotes: 0

Simon Sundarraj
Simon Sundarraj

Reputation: 1

Step 1 Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Step 2:

In \config\server.properties under Kafka folder Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

Step 3: In WLS2 Ubuntu,

sudo sysctl -p

After making above changes it worked for me in my local

Upvotes: 0

Vijay Chaudhari
Vijay Chaudhari

Reputation: 51

  1. Edit the file etc/sysctl.conf and add following lines in it.

    net.ipv6.conf.all.disable_ipv6=1

    net.ipv6.conf.default.disable_ipv6=1

  2. Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

  3. Update the sysctl config by using the following command. (Everytime you restart your machine this command needs to be run to update the configuration)

    sudo sysctl -p

Upvotes: 0

denise
denise

Reputation: 41

I got this problem when running a kafka producer in IntelliJ and a consumer in ubuntu terminal while on WSL2.

First, stop Kafka and Zookeeper. Then run these commands on WSL2, one by one:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

After that, in the kakfa folder, go to config/server.properties and edit the file to add the line:

listeners=PLAINTEXT://localhost:9092

When these commands have succeeded relaunch zookeeper and kafka.

https://www.conduktor.io/kafka/kafka-fundamentals

Upvotes: 3

Stephane Maarek
Stephane Maarek

Reputation: 5352

Stop Kafka and Zookeeper, then

Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Start Kafka, and you're good to go!

Upvotes: 9

shijin raj
shijin raj

Reputation: 91

I am able to find a work around . Thanks to Goose's comments

  1. I ran the following command in my WSL2 Ubuntu shell: ip addr
  2. Then ip address against the inet property global eth0 . for example, inet 172.20.XXX.XXX/20 .... scope global eth0
  3. I replaced all localhost with this IP address in the docker-compose.yml
  4. I replaced the localhost with this IP address in springboot yml or properties file.
  5. My Kafka producer and consumer able to connect to the Kafka running in Ubunti - WSL 2 from Windows

Upvotes: 2

Gautam
Gautam

Reputation: 3384

WSL2 runs on hypervisor and you need port proxy to connect Kafka Broker running on WSL2.

Step 1 . Check you WSL2 IP using following command and copy inet value

$ ifconfig
inet 172.X.X.X

Step 2. Open cmd with Admin permsissions

   netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

You should be able to connect now

Note : WSL2 IP changes everytime you restart machine

Upvotes: 5

Ed P
Ed P

Reputation: 421

Had this same issue. The root cause seems to be that WSL2 is broken with regards to IPv6 and localhost (See: https://github.com/microsoft/WSL/issues/4851)

The only fix I found that doesn't involve changing configs every time you reboot (per the "172.*" suggestion above) is to use the IPv6 loopback address ::1 in both the Kafka server config running in Linux and the Java client in Windows.

In server.properties I have this:

listeners=PLAINTEXT://[::1]:9092

And likewise in my Java client bootstrap server config I use

"[::1]:9092"

Upvotes: 32

Goose
Goose

Reputation: 141

I had the exact problem you are having and I resolved it as follows:

  1. I ran the following command in my WSL2 Ubuntu shell: ip addr | grep "eth0" I made note of the ip address against the inet property, for example, 172.27.10.68
  2. In my Kafka server.properties I replaced the listeners property value as follows: listeners=PLAINTEXT://172.27.10.68:9092 I commented out the advertised.listeners property. But you can alternatively assign the ip in question to this property, and have the listeners property set to 0.0.0.0. But I assume you are using the Kafka installation for testing/learning purposes, so I would keep it simple.
  3. I made no change to the Zookeeper's default ip:port
  4. I am using the Schema Registry, so I modified the Kafka bootstrap property as follows: kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092 I made no change to the default schema registry listener listeners=http://0.0.0.0:8081
  5. I used the same ip (as listed above) in my IntelliJ Kafka Producer. It then happily connected to my Kafka broker in WSL2.

More information on WSL2 networking can be found at https://learn.microsoft.com/en-us/windows/wsl/compare-versions .

The only problem with this setup is that every time you shutdown or restart your Windows machine, or close your Ubuntu terminal, the ip address for eth0 changes. And this results in redoing steps 2, 4 and 5. I am sure there is a better way, but everything I tried failed, except for this.

Upvotes: 14

g43m1
g43m1

Reputation: 543

This is not the optimal solution, but you will be able to connect if you run your producer in Ubuntu/WSL. This means if you are using a Windows IDE, writing the code, switching to Ubuntu and using a command line compiler and running the producer. See this post Error connecting to kafka server via IDE in WSL2

Upvotes: 0

Related Questions