CODEWITHSUNDEEP
elasticsearchlogstashapache-kafkaproducer-consumer
seba_sencha
seba_sencha

Reputation: 205

Prioritising message in Logstash and Kafka

We will be feeding different logs from different sources to kafka from logstash. There would be some particular set(or some format)messages that consumers of kafka will be more interested than other messages. How do we prioritise the messages?

One thought is to create two topics in kafka - 1) high_priority_topic 2) low_priority_topic and then use filter in logstash to push interested messages to high_priority_topic and rest of the messages to other. Is this possible?

Are there any recommended approach for this?

Upvotes: 1

Views: 152

Answers (1)

Will Barnwell
Will Barnwell

Reputation: 4089

Use either tags or type in logstash to differentiate between high_priority and low_priority, then have two kafka outputs.

if CONDITIONAL {
  kafka {
    topic => "high_priority_topic"
    ...
  }
}
else {
  kafka {
    topic => "low_priority_topic"
    ...
  }
}

Using type the CONDITIONAL would look like

[type] == "high_priority"

Using tagging it would look like

"high_priority" in [tags]

Upvotes: 2

Related Questions