CODEWITHSUNDEEP
scalaapache-kafka
sowen
sowen

Reputation: 1088

Kafka API: java.io.IOException: Can't resolve address: 357d78957cf5:9092

I am trying to figure out how to properly write a simple producer application to Kafka.

1) Please tell me if I should avoid using 0.10.0.1, I am totally struggled with it.

This example I found from Apache wiki worked for 0.8 https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example

It’s completely different in 0.10 http://kafka.apache.org/documentation#producerconfigs

So this is what I am using: "org.apache.kafka" % "kafka_2.10" % "0.10.0.1",

I am feeling very bad about this latest Kafka library, because apparently there are some known issues. In the runtime, it even shows a list of warnings about some configurations are not known config. But if I don’t provide them, the app won’t run. On github, people have identified this kind of issue.

2) Honestly, my code is super simple, if their documentation is even right. 

    val props = new Properties()
    props.put("bootstrap.servers", "192.168.99.100:9092");
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
    props.put("producer.type", "async")
    props.put("advertised.listeners", "192.168.99.100:32777")

    val producer = new KafkaProducer[String, String](props)
    val t = System.currentTimeMillis()
    val runtime = new Date().getTime();
    val msg = runtime + ", hello world";
    val message = new ProducerRecord[String, String](topic, null, msg)
    producer.send(message);

I am running 2 separate Docker containers for zookeeper (192.168.99.100:32777) and Kafka (192.168.99.100:9092). The settings work. I can use Kafka command line create a producer and consumer properly. However, trying to write my own client doesn’t work.

First, their document doesn’t even say advertised.listeners is required or part of the ProducerConfig. But without it, I encountered this exception “Give up sending metadata request since no node is available”. So I found this page: cann't connect to kafka server

Once I added “advertised.listeners” (because according to Apache document, “advertised.host.name” has been deprecated). I encountered this infinite exceptions: “java.io.IOException: Can't resolve address: 357d78957cf5:9092”

I honestly don’t know where that address come from. In the terminal, it clearly shows it’s using my docker container IP:

enter image description here

And then I found this: Kafka 0.8.2.2 - Unable to publish messages I think it’s the right answer to my problem, but I really don’t know what I am supposed to do to fix this problem.

Upvotes: 3

Views: 3922

Answers (2)

Benipal
Benipal

Reputation: 211

You have to add a host map.

In mac

vi /etc/hosts
{IP} {String in Error}
192.168.99.100 357d78957cf5

Now restart the application and test.

Upvotes: 0

amethystic
amethystic

Reputation: 7091

Some backgrounds you'll find useful before using the producer: The old producer(Scala) in 0.8.x had already been removed starting 0.9.0.

Therefore, you are using the new producer(Java) now. Kafka dev team decides to remove any Zookeeper dependencies from the new client, no matter producer or consumer. So you should not set the ZK setting anymore. Besides, letting "advertised.listeners" point to ZK service is incorrect.

"advertised.listeners" is for IaaS machines which usually have more than one NICs. Here is what Kafka doc says about it:

In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used.

You could set this parameter to have client bind the public NIC, whereas the intra-communication for brokers binds the private NIC.

Upvotes: 1

Related Questions