CODEWITHSUNDEEP
elasticsearchlogstashlogstash-configuration
Ryan
Ryan

Reputation: 10131

How to split a large json file input into different elastic search index?

The input to logstash is

input {
    file {
        path => "/tmp/very-large.json"
        type => "json"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }

and sample json file

{"type":"type1", "msg":"..."}
{"type":"type2", "msg":"..."}
{"type":"type1", "msg":"..."}
{"type":"type3", "msg":"..."}

Is it possible to make them feed into different elastic search index, so I can process them easier in the future?

I know if it is possible to assign them with a tag, then I can do something like

if "type1" in [tags] {
    elasticsearch {
        hosts => ["localhost:9200"]
        action => "index"
        index => "logstash-type1%{+YYYY.MM.dd}"
        flush_size => 50
    }
}

How to do similar thing by looking at a specific json field value, e.g. type in my above example?

Upvotes: 1

Views: 873

Answers (2)

baudsp
baudsp

Reputation: 4110

You can compare on any fields. You'll have to first parse your json with the json filter or codec.

Then you'll have a type field to work on, like this: 

if [type] == "type1" {
    elasticsearch {
        ...
        index => "logstash-type1%{+YYYY.MM.dd}"
    }
} else if [type] == "type2" {
    elasticsearch {
        ...
        index => "logstash-type2%{+YYYY.MM.dd}"
    }
} ...

Or like in Val's answer: 

elasticsearch {  
    hosts => ["localhost:9200"]  
    action => "index"  
    index => "logstash-%{type}%{+YYYY.MM.dd}"  
    flush_size => 50  
}

Upvotes: 0

Val
Val

Reputation: 217544

Even simpler, just use the type field to build the index name like this:

elasticsearch {
    hosts => ["localhost:9200"]
    action => "index"
    index => "logstash-%{type}%{+YYYY.MM.dd}"
    flush_size => 50
}

Upvotes: 1

Related Questions