CODEWITHSUNDEEP
elasticsearchlogstash
Killrazor
Killrazor

Reputation: 7176

How to send contents of a queue to elasticsearch index with logstash

I have a logstash up and running that consumes two rabbit queues and sends to an elasticsearch. This is my logstash.conf file:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
  }

}
filter{
}
output{
  stdout { codec => json}
    elasticsearch{
    hosts => ["elasticsearch"]
    index => "eventss-%{+YYYY.MM.dd}"
  }

}

Now I have another queue, but I want to send its content to a different elasticsearch index. My question is: how do I need to redirect specific entries to an specific index? Or do I need another logstash instance?

Thanks in advance.

Upvotes: 1

Views: 970

Answers (1)

Val
Val

Reputation: 217304

Very good start. Now you simply need to "type" each input and then forward the events to the appropriate output given its type, like this:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
    type => 'traces'               # <-- add this
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
    type => 'prices'               # <-- add this
  }

}
filter{
}
output{
  stdout { codec => json}

  if [type] == 'traces' {          # <-- check type
     elasticsearch{
       hosts => ["host1:9200"]
       index => "index1-%{+YYYY.MM.dd}"
     }
  }

  if [type] == 'prices' {          # <-- check type
     elasticsearch{
       hosts => ["host2:9200"]
       index => "index2-%{+YYYY.MM.dd}"
     }
  }
}

UPDATE

The above is the most general approach so that you can configure both outputs differently. As suggested by @pandaadb, you can also have a single output and define a type that would be your target index:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
    type => 'index1'                    # <-- add this
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
    type => 'index2'                    # <-- add this
  }

}
filter{
}
output{
  stdout { codec => json}

  elasticsearch{
    hosts => ["localhost:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"   # <-- use type here
  }
}

Upvotes: 3

Related Questions