Why can't I create a topic in Kafka?

Question

Following the Apache Kafka Quickstart Guide, I've done the following steps:

• Download and untar kafka_2.11-2.1.0.tgz
• cd kafka_2.11-2.1.0
• bin/zookeeper-server-start.sh config/zookeeper.properties
• In another terminal the the same dir, bin/kafka-server-start.sh config/server.properties
• bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test. Running echo $? afterward shows that this command exited with status 0

Now the moment of truth. The guide says:

We can now see that topic if we run the list topic command:

bin/kafka-topics.sh --list --zookeeper localhost:2181
> test

However, I get no output from that command, and a piece of software I'm testing, which tries to send a message on the "test" topic, crashes because it finds 0 partitions for the topic.

I also have a Ruby program that sends messages to Kafka on topic "test". It fails and retries, and I see the topic created at that point in the Kafka logs and can send messages to it. But even then, the command to list topics returns nothing.

Why can't I create a topic explicitly? Why can't I list topics that were created on demand? How can I troubleshoot this?

---

Logs

Here's what I see in the logs: https://gist.github.com/nathanl/bea7a45a056b2d44146947ec88c29185

Answer 1

1- first go to the Kafka directory and open the bin folder.

2- find the kafka-topics.sh file.

3- open it in notepad or any text editor application.

4- paste the following text in it:

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

exec $(dirname $0)/kafka-run-class.sh kafka.admin.TopicCommand "$@"

finally save and close the text editor. Done.

Answer 2

I faced this same issue on a mac while running on a VPN, which somehow caused my hostname resolution to fail.

I solved this by editing server.properties, adding the line:

listeners=PLAINTEXT://localhost:9092

Answer 3

I ran into the same issue.

First check if the bin/kafka-topics.sh has any content in it.

Then go ahead and clear all the tmp data of zookeeper and kafka

rm -r /tmp/zookeeper

rm -r /tmp/kafka-log

Then start your zookeeper and kafka again.

Answer 4

I faced to the same issue. However, I found out that the content of the script bin/kafka-topics.sh is empty somehow after the untar command.

So, I downloaded again the kafka, untar and check the content of the script file until I see it there, then it works.

Answer 5

On Mac 10.13.6, I followed the same steps, here are some troubleshooting steps for you to check.

When you start Zookeeper you should see in the stdout:

INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

When you start Kafka you should see in the stdout:

INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[…]
INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
INFO Kafka version : 2.1.0 (org.apache.kafka.common.utils.AppInfoParser)
INFO Kafka commitId : 809be928f1ae004e (org.apache.kafka.common.utils.AppInfoParser)
INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

The kafka-topics command should return output (you don't mention any, which could be a sign something didn't work at this stage):

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
$

When you do this, in the Kafka stdout you'll see Created log for partition test-0 amongst the log info:

INFO [Log partition=test-0, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
INFO [Log partition=test-0, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 52 ms (kafka.log.Log)
INFO Created log for partition test-0 in /tmp/kafka-logs with properties {compression.type -> producer, message.format.version -> 2.1-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
INFO [Partition test-0 broker=0] No checkpointed highwatermark is found for partition test-0 (kafka.cluster.Partition)
INFO Replica loaded for partition test-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO [Partition test-0 broker=0] test-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)

Topic then shows in the list:

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
test