CODEWITHSUNDEEP
apache-kafkadocker-compose
Dimitrije M
Dimitrije M

Reputation: 383

Unable to consume or produce to a remote Kafka broker

I have setup a simple droplet on Digital Ocean and am running a single Kafka and Zookeeper node which is started using a docker-compose file.

I am running into an issue with consuming or producing to the Kafka broker from outside of the Digital Ocean droplet. This is what my docker-compose looks like,

version: '3.4'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - /root/data/zookeeper/etc:/etc/zookeeper
      - /root/data/zookeeper/data:/var/lib/zookeeper/data
    container_name: "zookeeper"
    network_mode: "host"

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: PUBLIC_DIGITIAL_OCEAN_IP:2181
      KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://PUBLIC_DIGITIAL_OCEAN_IP:9093
      KAFKA_LISTENER: PUBLIC_DIGITIAL_OCEAN_IP:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=WARN"
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_TOOLS_LOG4J_LOGLEVEL: ERROR
    volumes:
      - /root/data/kafka/etc:/etc/kafka
      - /root/data/kafka/data:/var/lib/kafka/data
    container_name: "kafka"
    network_mode: "host"

I have tried different combinations with setting KAFKA_ADVERTISED_LISTENERS to use localhost, 0.0.0.0 and I am not having any success.

I can consume and produce if I enter the kafka container and use the CLI.

From what I have read, digital ocean does not have any firewall rules so the the ports are being exposed.

snippet from running netstat within the droplet

> netstat -tulpn | grep :2181
> tcp6       0      0 :::2181                 :::*                    LISTEN      10522/java  
> netstat -tulpn | grep :9093
> tcp6       0      0 :::9093                 :::*                    LISTEN      13093/java

Any help is greatly appreciated!

Upvotes: 1

Views: 482

Answers (1)

Dimitrije M
Dimitrije M

Reputation: 383

The issue was my firewall rules on the droplet. running the commands;

sudo ufw allow 2181 && sudo ufw allow 9092 resolved my issue.

Upvotes: 1

Related Questions