CODEWITHSUNDEEP
bashdockerapache-kafkaconfluent-platform
Joe
Joe

Reputation: 13111

Kafka inside Docker - how to read/write to a topic from command line?

I have a Kafka running inside Docker with SSL enabled.

docker ps:

CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                                        NAMES
b8f6b1c573a1        nginx:1.13.9-alpine                     "nginx -g 'daemon of…"   9 hours ago         Up 9 hours          80/tcp, 0.0.0.0:8081->443/tcp                ng
761ce6ee2960        confluentinc/cp-schema-registry:4.0.0   "/etc/confluent/dock…"   9 hours ago         Up 9 hours          0.0.0.0:8080->8080/tcp, 8081/tcp             sr
16d7b81dfbc8        confluentinc/cp-kafka:4.0.0             "/etc/confluent/dock…"   9 hours ago         Up 9 hours          0.0.0.0:9092-9093->9092-9093/tcp             k1
9be579992536        confluentinc/cp-zookeeper:4.0.0         "/etc/confluent/dock…"   9 hours ago         Up 9 hours          2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zk

How to write to a topic from command line?
Tried (topic 'test' exists):

kafka-console-producer  --broker-list kafka:9093 --topic test
# [2018-04-23 17:55:14,325] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
# org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. (60s timeout)


kafka-console-producer  --broker-list kafka:9092 --topic test
>aa
#[2018-04-23 18:00:59,443] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
#[2018-04-23 18:00:59,444] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
# org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]

kafka-console-producer  --broker-list localhost:9092 --topic test                                                                                                                                                             dnk306@9801a7a5b33d
>aa
#[2018-04-23 21:52:47,056] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
#[2018-04-23 21:52:47,056] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
#org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]

Upvotes: 4

Views: 13983

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191844

When inside the schema registry or Kafka containers, and if you used the Docker compose configuration, this is possible as the containers will be linked by Compose, however, your container hostname is likely k1 and not kafka

If you are inside the Kafka container or using CLI commands outside the container then it's simply localhost:9093 (because you forwarded the port).

If you are inside some container other than the Kafka one and you want to resolve the Kafka container by hostname, you must add --link kafka at docker run. See Docker documentation for linking containers

Also important, you'll need to link the Kafka container to the Zookeeper container, and Schema Registry to one or the other ZK or Kafka, depending on how it's configured.

https://docs.confluent.io/current/installation/docker/docs/quickstart.html

Also SSL Kafka Docker compose examples here

TopicAuthorizationException: Not authorized to access topics: [test]

This indicates a successful connection to the Kafka container. You're next step is to ensure your Java environment has the necessary keys to access the broker over SSL

Upvotes: 1

Related Questions