CODEWITHSUNDEEP
templateselasticsearchlogstash
Vagif Abilov
Vagif Abilov

Reputation: 9991

Logstash keeps overwriting Elasticsearch index template

I am sending log data to Elasticsearch database using Logstash. I wanted to change the number of shards from 3 to 1 and issued the following command via ES REST API:

PUT server_name/_template/logstash
{
    "template": "logstash",
        "settings": {
            "index.number_of_replicas": "0",
            "index.refresh_interval": "5s",
            "index.number_of_shards": "1"
        }
}

The server responsed OK and if I issue GET _template/logstash I can see that the number of shards is now set to 1.

Then I start logstash with an output set to ship logs to Elasticsearch. There are not template-related settings. After I send log data I see that the number of shards is set back to its default value (3).

I even tried to override it by referring template from Logstash configuration file. Nope, whatever I specify the settings are reset back. It looks like Logstash keeps on overwriting Elasticsearch index settings with some defaults, and I can't figure out how to disable this.

UPDATE. I've added the following lines to the Logstash config file but it didn't help:

    manage_template => false
    template_overwrite => true

Also tried template_overwrite set to false. And I tried two different ways of setting number of shards in the JSON file:

{
    "logstash": {
        "template": "logstash-*",
        "settings": {
            "index.number_of_replicas": "0",
            "index.refresh_interval": "5s",
            "index.number_of_shards": "1"
        }
    }
}

and

{
    "template": "logstash-*",
    "settings" : {
        "index.number_of_shards" : 1,
        "index.number_of_replicas" : 0,
    }
}

Upvotes: 1

Views: 6317

Answers (2)

Vagif Abilov
Vagif Abilov

Reputation: 9991

OK, after many hours I've found the following:

  1. The main reason for overwriting Logstash template was that I had two Elasticsearch parts in Logstash output section with conditional selection (if/else). Even though I correctly configured index settings in the first ("if") part, as soon as Logstash encountered logs that satisfied the other ("else") part, it used default template that overwrote the custom one.

  2. "manage_template" option is important. It has to be set to "true" for the configuration block that refers to the custom template settings file, and "false" for the block that shouldn't overwrite the custom Logstash template.

Upvotes: 1

Alcanzar
Alcanzar

Reputation: 17155

On you elasticsearch {} element in your Logstash configuration, you need to add manage_template => false if you want to manage the template outside of logstash.

Upvotes: 1

Related Questions