CODEWITHSUNDEEP
dockercontainerslogstashapache-kafkadocker-image
Gman
Gman

Reputation: 2453

Docker: cant send data from logstash container to Kafka container

I have 2 docker containers, 1 running Logstash and the other running Zookeeper and Kafka. I am trying to send data from Logstash to Kafka but can't seem to get data across to my topic in Kafka.

I can log into the Docker Kafka container and produce a message to my topic from the terminal and then also consume it.

I am using the output kafka plugin:

output {
    kafka {
        topic_id => "MyTopicName"
        broker_list => "kafkaIPAddress:9092"
    }
}

The ipAddress I got from running docker inspect kafka2

When I run ./bin/logstash agent --config /etc/logstash/conf.d/01-input.conf I get this error.

Settings: Default pipeline workers: 4
Unknown setting 'broker_list' for kafka {:level=>:error}
Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Something is wrong with your configuration.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/config/mixin.rb:134:in `config_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/outputs/base.rb:63:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/output_delegator.rb:74:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/agent.rb:473:in `start_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}

I have checker the configuration of the file by running the following command which returned OK.

 ./bin/logstash agent --configtest --config /etc/logstash/conf.d/01-input.conf
Configuration OK

Has anyone ever come across this, is it that I haven't opened the ports on the kafka container and if so how can I do that while keeping Kafka running?

Upvotes: 1

Views: 624

Answers (1)

martyc
martyc

Reputation: 26

The error is here broker_list => "kafkaIPAddress:9092"

try bootstrap_servers => "KafkaIPAddress:9092" if you have the containers on separate machines, map kafka to the host 9092 and use the host address:port, if on same host use the internal Docker IP:port

Upvotes: 1

Related Questions