CODEWITHSUNDEEP
apache-kafkastrimzi
lipsum
lipsum

Reputation: 969

Kafka topic not found when running strimzi on docker-desktop k8s cluster

I try to run strimzi on docker-desktop k8s, following this document.

I set up all the things.

$ kubectl get all -n my-kafka-project                                           
NAME                                              READY   STATUS    RESTARTS   AGE
pod/my-cluster-entity-operator-7ddb6d5b88-q2xg8   3/3     Running   0          97m
pod/my-cluster-kafka-0                            1/1     Running   0          98m
pod/my-cluster-zookeeper-0                        1/1     Running   0          98m

NAME                                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/my-cluster-kafka-0                    NodePort    10.99.207.179    <none>        9094:31805/TCP               98m
service/my-cluster-kafka-bootstrap            ClusterIP   10.103.193.53    <none>        9091/TCP,9092/TCP,9093/TCP   98m
service/my-cluster-kafka-brokers              ClusterIP   None             <none>        9091/TCP,9092/TCP,9093/TCP   98m
service/my-cluster-kafka-external-bootstrap   NodePort    10.97.198.62     <none>        9094:31314/TCP               98m
service/my-cluster-zookeeper-client           ClusterIP   10.101.206.203   <none>        2181/TCP                     98m
service/my-cluster-zookeeper-nodes            ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   98m

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-cluster-entity-operator   1/1     1            1           97m

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/my-cluster-entity-operator-7ddb6d5b88   1         1         1       97m

NAME                                    READY   AGE
statefulset.apps/my-cluster-kafka       1/1     98m
statefulset.apps/my-cluster-zookeeper   1/1     98m

And get nodeport:

$ kubectl get service my-cluster-kafka-external-bootstrap -n my-kafka-project -o=jsonpath='{.spec.ports[0].nodePort}{"\n"}'
31314

While this document uses minikube, I can access docker-desktop node by localhost, so I try to access localhost:31314, but I can't produce message to topic. It seems I can successfully connect to broker, but can't to topic.

./kafka-console-producer.sh --broker-list localhost:31314 --topic my-topic 
>Test
[2021-05-15 15:41:13,681] ERROR Error when sending message to topic my-topic with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Topic my-topic not present in metadata after 60000 ms.

I've checked with kafkacat, and topic itself is created successfully.

kafkacat -b localhost:31314 -L
Metadata for all topics (from broker -1: localhost:31314/bootstrap):
 1 brokers:
  broker 0 at 192.168.65.4:31805 (controller)
 4 topics:
  topic "__strimzi_store_topic" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "my-topic" with 3 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 1, leader 0, replicas: 0, isrs: 0
    partition 2, leader 0, replicas: 0, isrs: 0
  topic "__strimzi-topic-operator-kstreams-topic-store-changelog" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "__consumer_offsets" with 50 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 1, leader 0, replicas: 0, isrs: 0
    partition 2, leader 0, replicas: 0, isrs: 0
    partition 3, leader 0, replicas: 0, isrs: 0
    partition 4, leader 0, replicas: 0, isrs: 0
    partition 5, leader 0, replicas: 0, isrs: 0
    ...

I don't fully understand mechanism how the message is sent to topic, so which point should I check next?

Upvotes: 2

Views: 1664

Answers (1)

lipsum
lipsum

Reputation: 969

Based on Jakub's comment, I can produce and consume message from host by changing Kafka resource in my-kafka-project namespace.

I've overridden the advertised listeners in the external listener of .spec.kafka.listeners.

from:

- name: external
  port: 9094
  tls: false
  type: nodeport

to:

- name: external
  port: 9094
  tls: false
  type: nodeport
  configuration:
    brokers:
    - broker: 0
      advertisedHost: localhost
      advertisedPort: 31314

Upvotes: 4

Related Questions