CODEWITHSUNDEEP
elasticsearchlogstashlogstash-configurationfilebeat
Smoeey
Smoeey

Reputation: 127

Use Logstash "%{[source]}" to show the file name only for the Elasticsearch Index

I would like to use the filename from the source file as the index for my elasticsearch entries as we will have multiple different log files logging to Elasticsearch using FileBeats and LogStash.

Currently I have:

input
{
  beats {
    port => 5044
  }
}

filter {
    json {
    source => "message"
    }   
}

output {
    elasticsearch {
        hosts => "localhost:9200"
        manage_template => false
        index => "%{[source]}"
        document_type => "%{[@metadata][type]}"
        user => ***
        password => ***
    }
}

This provides me with "C:\logs\test-20170518.json". I would like to have test-20170518 used as the index only. Can this be done using the source?

Upvotes: 0

Views: 2801

Answers (1)

berrytchaks
berrytchaks

Reputation: 849

You can used a grok filter plugin for that. Try this

input
{
  beats {
    port => 5044
  }
}

filter {
    json {
        source => "message"
    }  
    grok {
        match => [
            "source",
            "C:\\logs\\%{DATA:myIndex}.json"
      ]
    } 
}


output {
    elasticsearch {
        hosts => "localhost:9200"
        manage_template => false
        index => "%{[myIndex]}"
        document_type => "%{[@metadata][type]}"
        user => ***
        password => ***
    }
}

Upvotes: 3

Related Questions