CODEWITHSUNDEEP
jsonelasticsearchlogstash
Michal Fašánek
Michal Fašánek

Reputation: 513

How to setup Logstash config target index?

I am trying to set the target elasticsearch index dynamically in Logstash configuration file by a document field. Unfortunately, it does not seem like logstash is interpreting it like a variable, but rather a string.

It managed to create a lot of new indices as I have been trying to reach the index name in document.

enter image description here

which roughly summarizes my attempts. Here is an example of message I outputted into a file:

{  
"tags":[],
"type":"logstash",
"stack_info":null,
"level":"INFO",
"@timestamp":"2018-02-12T13:30:34.332Z",
"@version":"1",
"host":"Michals-MacBook-Pro.local",
"logger_name":"python-logstash-logger",
"message":"{
    \"index_name\": \"THIS_IS_MY_INDEX_NAME\"
    }",
"path":"/Users/michalfasanek/..."
}

This is currently my config file: 

input { 
    udp { 
            port => 5959 
            codec => json  
    } 
} 

filter {

}

output {


    elasticsearch { 

            hosts => "localhost:9200"
            index => "[message][index_name]"
    }

    file {
            path => "/usr/local/Cellar/logstash/6.2.0/controloutputnay"
    }

    stdout { }
}

How can I refer to it as a variable and not as a string? Thanks for any suggestions.

Upvotes: 1

Views: 1981

Answers (2)

baudsp
baudsp

Reputation: 4110

The message field in a logstash event is a string, so you'll have to extract the index_name from the message string field into another field, then use it in your elasticsearch output plugin.

You can use the grok filter plugin to do so:

grok { 
 match => ["message", "\"index_name\": \"%{WORD:index_name}\""] 
}

This filter will create a field called index_name. Then you can use it in your output:

elasticsearch { 
   hosts => "localhost:9200"
   index => "%{index_name}"
}

Upvotes: 1

Michal Fašánek
Michal Fašánek

Reputation: 513

I solved this myself, but not exactly the way I wanted. I discovered the "extra" parameter in the Python logger I am using to generate documents.

https://pypi.python.org/pypi/python-logstash

This "extra" are fields added to the top-level of JSON, which means I do not have to deal with 'index_name' field being nested inside 'message' field either.

After this, my config line is simply 

index => "%{index_name}"

While this works perfectly for me, including not having to filter out 'index_name' field after using it, I would still like to know why it did not work in the first place. Any suggestions are welcome.

Upvotes: 1

Related Questions