CODEWITHSUNDEEP
logstash
Ido Barash
Ido Barash

Reputation: 5122

logstash output to kafka - topic data in message

I want to create a conf file for logstash that loads data from a file and send it to kafka.

the file is in json format and has the topicId in it.

This is what I have so far..

input {
   file {
    path => "~/file1.json"
    start_position => "beginning"
    codec => "json"
  }
}
filter {
    json {
        source => message
    }
}
output {
  kafka { 
    bootstrap_servers => "localhost"
    codec => plain {
        format => "%{message}"
    }
    topic_id =>  "???"
  }
}

can this be done?

Regards, Ido

Upvotes: 2

Views: 9550

Answers (1)

Arijeet Saha
Arijeet Saha

Reputation: 1168

Yes it can be done.

For example if the message json contains a topic_id key like:

"topicId": "topic1"

Then in logstash kafka output plugin:

output {
  kafka { 
    bootstrap_servers => "localhost"
    codec => plain {
        format => "%{message}"
    }
    topic_id =>  "%{topicId}"
  }
}

Upvotes: 3

Related Questions