CODEWITHSUNDEEP
dockerapache-kafkadocker-composelogstash
Mandir Vaibhav
Mandir Vaibhav

Reputation: 33

Logstash not receiving data from kafka running inside docker(compose) using port other than 9092

I want to use kafka with elk stack, where filebeat reads logs from file and send data to kafka which then sends it to logstash. I am running kafka using docker-compose file.

I managed to run it by using port 9092 which is default for kafka, but problem is my organization is already running some other service on port 9092. So I have used 9095 in my docker-compose for host and 9092 inside container.

When I try to use filebeat and logstash with kafka, data is not getting to logstash. I can see that logstash is subscribed to topic I created, but it is not getting any data even when I send message from console-producer. But when I use console-producer and console-consumer it is working fine. But it is not working with logstash and filebeat which are running outside the container.

Here is my docker-compose file for kafka:

version: "3"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
    - 2181:2181

 

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
    - 9095:9092
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
      KAFKA_ADVERTISED_LISTENERS: INSIDE://172.18.0.3:9092,OUTSIDE://192.168.135.57:9095
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

Here 192.168.135.57 is my host address(public IP of my machine) and 172.18.0.3 is address of docker container running kafka. and this is my logstash config file:

input {
  kafka
    {
            bootstrap_servers => "192.168.135.57:9095"
            topics => "topicname"
    }
}

output {
    stdout{codec => rubydebug}
    elasticsearch {
        hosts =>  "http://localhost:9200"
        index => "topic-test"    
    }
}

This is what I used in filebeat to send data to kafka:

output.kafka:
  hosts: ["192.168.135.57:9095"]
  topic: "topicname"

I can't figure out what is wrong with this pipeline. I would like to run this docker-compose on port 9095(or any other port except 9092) on host machine. Help would be appreciated.

Upvotes: 0

Views: 433

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191844

You're still mapping 9095 on the host to the advertised INSIDE listener

You should use 9095:9095 to map to the OUTSIDE listener if that is what other clients will connect on

Upvotes: 1

Related Questions