CODEWITHSUNDEEP
elasticsearchlogstashelastic-stacklogstash-grok
red888
red888

Reputation: 31560

Only allow fields that are in the index template

I have logstash pushing docs into an elasticsearch cluster.

And I apply a template to the indices with logstash:

elasticsearch {
  hosts => 1.1.1.1.,2.2.2.2.
  index => "logstash-myindex-%{+YYYY-MM-dd}"
  template_name => "mytemplate"
  template => "/etc/logstash/index_templates/mytemplate.json"
  template_overwrite => true
}

Is there a way I can have only the fields defined in the template get added to the docs? Because sometimes the docs have a bunch of other fields I don't care about and I don't want to manually filter out each one. I want to be able to say if field not in index template do not add.

edit: I did this in my index template but fields not specified in the template are still getting added to docs:

{
  "template": "logstash-myindex*",
  "order": 10,
  "mappings": {
    "_default_": {
      "dynamic": "scrict",
      "_all": {
        "enabled": false
      },
      "properties": {
        "@timestamp": {
          "type": "date",
          "include_in_all": false
        },
        "@version": {
          "type": "keyword",
          "include_in_all": false
        },
        "bytesReceived": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
         .... etc

Upvotes: 0

Views: 140

Answers (1)

Sperr
Sperr

Reputation: 599

I'm not familiar with logstash - but I'm assuming this is just like creating an index in ElasticSearch.

In ElasticSearch you can disabled the dynamic creation of fields by adding:

"dynamic": false

to the mapping.

This would look something like this:

{
  "mappings": {
    "_default_": { 
      "dynamic": false
    }
  }
}

Upvotes: 1

Related Questions