CODEWITHSUNDEEP
phpdockerapache-kafka
Horse
Horse

Reputation: 3063

Failed to resolve 'kafka:9092': Name or service not known - docker / php-rdkafka

I am trying to get php connecting to kafka all within a docker container.

kafka php lib - https://github.com/arnaud-lb/php-rdkafka/

kafka docker container - https://hub.docker.com/r/wurstmeister/kafka/

Everything is building and running ok, but when I try to connect with the PHP producer, I get the following:

httpd_1   | %3|1490816385.542|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Name or service not known
httpd_1   | %3|1490816385.543|ERROR|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Name or service not known
httpd_1   | %3|1490816385.543|ERROR|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down

I am using the following in PHP

    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers("kafka");

    $topicConf = new RdKafka\TopicConf();
    $topicConf->set("message.timeout.ms", 1000);
    $topic = $rk->newTopic("DEV", $topicConf);

    $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message");

    $rk->poll(1000);
    $kafkaConf = new RdKafka\Conf();
    $kafkaConf->setErrorCb(function ($rk, $err, $reason) {
        printf("Kafka error: %s (reason: %s)\n", rd_kafka_err2str($err), $reason);
    });
    $kafkaConf->setDrMsgCb(function ($rk, $message) {
        if ($message->err) {
            print_r($message);
        } else {
            print_r("ok");
        }
    });

I've played around, trying to set the host IP in both the docker-compose.yml and the PHP code, but no joy. I also had connection refused but I don't know if thats better or worse?

If it helps, my docker-compose.yml

httpd:
  build: .
  ports:
  - 8180:80
  volumes:
  - ~/www:/var/www/html
zookeeper:
  image: wurstmeister/zookeeper
  ports:
  - "2181:2181"
kafka:
  build: ~/kafka-docker/.
  links:
  - zookeeper
  ports:
  - "9092:9092"
  environment:
    KAFKA_ADVERTISED_HOST_NAME: kafka
    KAFKA_ADVERTISED_PORT: 9092
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
  - /var/run/docker.sock:/var/run/docker.sock

My question: Any idea how I can connect to kafka successfully from php?

Upvotes: 21

Views: 46531

Answers (5)

user1519240
user1519240

Reputation: 2354

I was able to access Kafka running in Docker with the following docker-compose.yml file:

version: '2'

services:
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
      - '9093:9093'
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - KAFKA_ADVERTISED_HOST_NAME=kafka

KAFKA_ADVERTISED_LISTENERS and KAFKA_ADVERTISED_HOST_NAME made the trick!

Then ensure you have 127.0.0.1 kafka in the /etc/hosts file

Upvotes: 1

Shady Smaoui
Shady Smaoui

Reputation: 1295

vim /etc/hosts

Then add 127.0.0.1 kafka to your /etc/hosts

Upvotes: -1

Bhala T R
Bhala T R

Reputation: 80

I have added the following in the hosts file of the OS (on windows - C:\Windows\System32\drivers\Etc\hosts & on linux - vi /etc/hosts): kafka Then could see the kafka messages coming through from that machine to my machine locally.

Upvotes: -1

张馆长
张馆长

Reputation: 1859

if containers' host IP is 192.168.1.110,could set it in docker container env:

KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://192.168.1.110:{{ kafka_port }}"

Upvotes: 10

Edenhill
Edenhill

Reputation: 3113

The brokers will advertise themselve using advertised.listeners (which seems to be abstracted with KAFKA_ADVERTISED_HOST_NAME in that docker image) and the clients will consequently try to connect to these advertised hosts and ports.

You will thus need to make sure the client can resolve and reach this advertised hostnames, e.g. by adding "kafka" to /etc/hosts on the client host.

Upvotes: 38

Related Questions