CODEWITHSUNDEEP
configurationcluster-computingapache-kafka
Eddy Kim
Eddy Kim

Reputation: 61

kafka cluster configuration

I'm new on Kafka. I have question about kafka configuration.

I want to using seperate server like below,

server1: kafka producer server2: kafka broker, kafka consumer, zookeeper

But, I can't send message to broker. And I got this error messages.

on console-producer(server1), console stdout error message `

[2016-05-24 16:41:11,823] ERROR Error when sending message to topic twitter with key: null, value: 3 bytes with error: Failed to update metadata after 60000 ms.(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

`

on kafka producer(server2), console stdout error message `

[2016-05-25 10:20:01,588] DEBUG Connection with /192.168.50.142 disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:160)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:141)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
    at kafka.network.Processor.run(SocketServer.scala:413)
    at java.lang.Thread.run(Thread.java:745)

`

running commands are like below

server1 on kafka dir, `

./bin/zookeeper-server-start.sh config/zookeeper.properties
./bin/kafka-server-start.sh config/server.properties
./bin/kafka-console-consumer.sh --zookeeper 192.168.50.142:2181 --from-beginning --topic twitter
./bin/kafka-topics.sh --create --zookeeper 192.168.50.142:2181 --replication-factor 1 --partitions 1 --topic twitter

`

and server2 on kafka dir, `

./bin/kafka-console-producer.sh --broker-list 192.168.50.142:9092 --topic twitter

`

And my configuration are,

server1(IP: 192.168.50.155):

kafka/config/producer.properties `

metadata.broker.list=192.168.50.142:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder

`

server2(IP:192.168.50.142):

kafka/config/zookeeper.properties `

dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0

`

kafka/config/server.properties `

broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
broker.id=0
port=9092
log.dir=/tmp/kafka-logs-1
delete.topic.enable=true

`

kafka.config/consumer.properties `

zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group

`

kafka_2.11-0.9.0.0 java 1.8.0_60 node v4.4.4

Should I need to change any configuration? Please give some help.

Upvotes: 6

Views: 4803

Answers (3)

Chandan
Chandan

Reputation: 166

Make the following changes in server.properties:

  1. Change the second line to 

    listeners=PLAINTEXT://:9092

  2. Add this line:

    advertised.listeners=PLAINTEXT://192.168.50.142:9092

    This line is needed because this is the hostname and port the broker will advertise to consumers and producers. And since you have a producer on another machine, this line is needed.

While writing any command in the terminal, use 

    <command> --zookeeper localhost:2181 <rest of it>

Hope this works.

Upvotes: 1

Ehasanul Hoque
Ehasanul Hoque

Reputation: 650

You need to modify server.properties file with appropriate config value and to update /etc/hosts file for resolving your machine to IP.

Upvotes: 0

Kamal Chandraprakash
Kamal Chandraprakash

Reputation: 2002

It seems your producer configurations are not correct.

kafka/config/producer.properties 

bootstrap.servers=192.168.50.142:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

Upvotes: 1

Related Questions