CODEWITHSUNDEEP
elasticsearchlogstashaws-elasticsearch
sedavidw
sedavidw

Reputation: 11691

Elasticsearch index not being created with settings from logstash template

I have a bulk upload for a new index that I'm sending to my ES cluster from logstash. As such I want replication and refreshing turned off until the load is done, and I'll re-enable those values after the upload is complete.

I have a config file that looks like the following

input {
      stdin { type => stdin }
}
filter {
  csv {
     separator => " "
     columns => [ ...]
  }
}

output {
   amazon_es {
     hosts =>
         ["my-domain.us-east-1.es.amazonaws.com"]
     index => "my-index"
     template => "conf/my-index-template.json"
     template_name => "my-index-template-name"
     region => "us-east-1"
  }
}

And the template file looks like

{
    "template" : "my-index-template-name",
    "mappings" : {
      ...
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "48",
        "number_of_replicas" : "0",
        "refresh_interval": "-1"
      }
    }
}

And when I run logstash and go to look at the settings for that index, the mappings are all respected from this template which is good, but everything in the settings section is ignored and it takes on default values (i.e. number_of_shards=5, and number_of_replicas=1)

Some investigation notes: If I get the template after it's installed from ES itself I see the proper values in the template (for both mappings and settings). They just don't seem to be applying to the index

Also if I take the contents of the template file and create the index manually w/ a PUT it shows up as I would expect

My logstash version is 7.3.0 and my elasticsearch version is 6.7

Not sure what I'm doing wrong here

Upvotes: 0

Views: 1720

Answers (1)

leandrojmp
leandrojmp

Reputation: 7463

Your index name is my-index, but the template setting in your mapping uses my-index-template-name, it needs to be a regular expression or the same name as your index.

Since you are using elasticsearch 6.7 you should use index_patterns instead of template in your mapping.

{
    "index_patterns" : ["my-index"],
    "mappings" : {
      ...
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "48",
        "number_of_replicas" : "0",
        "refresh_interval": "-1"
      }
    }
}

Upvotes: 2

Related Questions