CODEWITHSUNDEEP
jsonazureelasticsearchlogginglogstash
bollicino
bollicino

Reputation: 75

logstash output elasticsearch - generate index name dynamically

I'm working with elastic and logstash versione 6.2.4.

Logstash input is configured to read data from Azure EventHubs

input
{
    azureeventhub
    {
        key => ""
        username => "ReadAccess"
        namespace => "myeventhubs"
        eventhub => "logstash"
        partitions => 2
        consumer_group => "logstash-cg"
    }
}

My events are a JSON message, like this

{
    "log": {
      "event": "....."
    },
    "header": {
      "remoteName": "foobar"
    }
}

I need to create an index in elasticsearch for every remoteName, so I tried to configure output to elastic like this:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
     index => "log-%{header.remoteName}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

I expected to find an index named: log-foobar-2018-04-22. But didn't work.

In Elasticsearch I found a single index named: log-%{header.remoteName}-2018-04-22

Is it possibile create index dynamically? How I need to configure logstash input?

Upvotes: 3

Views: 5010

Answers (1)

malle
malle

Reputation: 56

Try following:

index => "log-%{[header][remoteName]}-%{+YYYY.MM.dd}"

Field References

It is often useful to be able to refer to a field by name. To do this, you can use the Logstash field reference syntax.

The syntax to access a field is [fieldname]. If you are referring to a top-level field, you can omit the [] and simply use fieldname. To refer to a nested field, you specify the full path to that field: [top-level field][nested field].

https://www.elastic.co/guide/en/logstash/5.2/event-dependent-configuration.html

Upvotes: 4

Related Questions