CODEWITHSUNDEEP
elasticsearchlogstash
Orkun
Orkun

Reputation: 522

logstash output with document oriented elasticsearch

First question about this topic is with the code below, we can pass logstash output to elasticsearch. As you know, elasticsearch is document oriented. But the code below, we can only define an index not a document id. I want to ask that is it possible to define different document id for every logstash pipeline output ? If your answer is yes how, if not why ?

Second question is, if i run the code below. I have got yellow state in elasticsearch node. I know how to change it with manuely or sending dsl query via python etc. But is it possible to solve this inside of below code ?

elasticsearch {
        hosts => ["localhost:9200"]
        index => "name_of_index" 
        http_compression => true
    }

Thanks for answering..

Upvotes: 0

Views: 339

Answers (1)

Val
Val

Reputation: 217514

You can definitely specify the document ID using the document_id setting:

elasticsearch {
    hosts => ["localhost:9200"]
    index => "name_of_index-%{+YYYY.MM.dd}"
    manage_template => true
    template_name => "my-template"
    template => "/path/to/my-template.json"
    document_id => "%{my_id_field}" 
    http_compression => true
}

You can also make sure that every day a new index is created by specifying the date pattern in the index name (see above).

Also make sure to have the following in a file called my-template.json that is referenced in the elasticsearch output. Its role is to provide the specific settings to use when creating your index. Here, since you have a single node, we're instructing the index to not create any replica shards, to make sure that the cluster will be green.

my-template.json

{
  "index_patterns": ["name_of_index*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }
}

Upvotes: 1

Related Questions