CODEWITHSUNDEEP
apache-kafkalogstash
Darren
Darren

Reputation: 792

Is there a way to determine logstash input plugin from message?

I have two (kafka) input plugins in my logstash config. Each input plugin is configured to listen to a different kafka broker. I would like to include in the message some field that represents which input plugin generated the message. I was hoping to be able to include a hard-coded field like "input=kafka1", "input=kafka2", etc. in my input definition. I don't think I can use mutate filter because by that time, I don't know where the message came from?

Upvotes: 0

Views: 241

Answers (1)

leandrojmp
leandrojmp

Reputation: 7463

Use the tags option in your inputs.

input {
    kafka {
        *** your first input config ***
        tags => ["kafka1"]
    }
    kafka {
        *** your second config ***
        tags => ["kafka2"]
    }
}

Then you can use those tags to filter in your filter block.

filter {
    if "kafka1" in [tags] {
        filters for kafka 1 tag
    }

    if "kafka2" in [tags] {
        filters for kafka 2 tag
    }
}

You can use the same conditionals in your output block.

You can also use multiple pipelines and have a different pipeline for each kafka broker.

Upvotes: 1

Related Questions