CODEWITHSUNDEEP
elasticsearch
Volodymyr
Volodymyr

Reputation: 285

Disable dynamic mapping creation for only specific indexes on elasticsearch?

I'm trying to disable dynamic mapping creation for only specific indexes, not for all. For some reason I can't put default mapping with 'dynamic' : 'false'. So, here left two options as I can see:

  1. specify property 'index.mapper.dynamic' in file elasticsearch.yml.
  2. put 'index.mapper.dynamic' at index creation time, as described here https://www.elastic.co/guide/en/kibana/current/setup.html#kibana-dynamic-mapping

First option may only accept values: true, false and strict. So there is no way to specify subset of specific indexes (like we do by pattern with property 'action.auto_create_index' https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation).

Second option just not works. I've created index

POST http://localhost:9200/test_idx/
{
    "settings" : {
        "mapper" : {
            "dynamic" : false
        }
    },
    "mappings" : {
        "test_type" : {
            "properties" : {
                "field1" : {
                    "type" : "string"
                }
            }
        }
    }
}

Then checked index settings:

GET http://localhost:9200/test_idx/_settings    
{
    "test_idx" : {
        "settings" : {
            "index" : {
                "mapper" : {
                    "dynamic" : "false"
                },
                "creation_date" : "1445440252221",
                "number_of_shards" : "1",
                "number_of_replicas" : "0",
                "version" : {
                    "created" : "1050299"
                },
                "uuid" : "5QSYSYoORNqCXtdYn51XfA"
            }
        }
    }
}

and mapping:

GET http://localhost:9200/test_idx/_mapping
{
    "test_idx" : {
        "mappings" : {
            "test_type" : {
                "properties" : {
                    "field1" : {
                        "type" : "string"
                    }
                }
            }
        }
    }
}

so far so good, let's index document with undeclared field:

POST http://localhost:9200/test_idx/test_type/1
{
    "field1" : "it's ok, field must be in mapping and in source",
    "somefield" : "but this field must be in source only, not in mapping"
}

Then I've checked mapping again:

GET http://localhost:9200/test_idx/_mapping
{
    "test_idx" : {
        "mappings" : {
            "test_type" : {
                "properties" : {
                    "field1" : {
                        "type" : "string"
                    },
                    "somefield" : {
                        "type" : "string"
                    }
                }
            }
        }
    }
}

As you can see, mapping is extended regardless of index setting "dynamic" : false. I've also tried to create index exactly as described in doc 

PUT http://localhost:9200/test_idx
{
    "index.mapper.dynamic": false
}

but got the same behavior.

Maybe I've missed something?

Thanks a lot in advance!

Upvotes: 22

Views: 26965

Answers (6)

Rafiq
Rafiq

Reputation: 11465

first, please be concern aboout value false or strict,they work in a different way.

using "dynamic": "false" and create documents with fields not covered by the mapping, those fields will be ignored (so they won't be stored) and wouldn't show up in _source when you GET the document.

where value strict will not allow you to create the document rather it will throw an exception

Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.

However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.

https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html

PUT my-index-000001
{
  "mappings": {
    "dynamic": false, 
    "properties": {
      "user": { 
        "properties": {
          "name": {
            "type": "text"
          },
          "social_networks": {
            "dynamic": true, 
            "properties": {}
          }
        }
      }
    }
  }
}

if you are using node.js client

    await this.client.indices.putMapping({
      index: ElasticIndex.UserDataFactory,
      body: {
        dynamic: 'strict',
        properties: {
          ...this.schema,
        },
      },
    });

Upvotes: 1

SoH
SoH

Reputation: 2190

For ES 7 if you want to update an existing index:

PUT customers/_mapping
{
  "dynamic": "strict"
}

Upvotes: 0

avetisk
avetisk

Reputation: 12319

The answer is in the doc (7x.): https://www.elastic.co/guide/en/elasticsearch/reference/7.x/dynamic.html

The dynamic setting controls whether new fields can be added dynamically or not. It accepts three settings:

true

Newly detected fields are added to the mapping. (default)

false

Newly detected fields are ignored. These fields will not be indexed so will not be searchable but will still appear in the _source field of returned hits. These fields will not be added to the mapping, new fields must be added explicitly.

strict

If new fields are detected, an exception is thrown and the document is rejected. New fields must be explicitly added to the mapping.

PUT my_index
{
  "mappings": {
    "dynamic": "strict", 
    "properties": {
      "user": { 
        "properties": {
          "name": {
            "type": "text"
          },
          "social_networks": { 
            "dynamic": true,
            "properties": {}
          }
        }
      }
    }
  }
}

Upvotes: 3

Antoine Vulcain
Antoine Vulcain

Reputation: 44

You cannot disable dynamic mapping in ES 7 anymore, what you can do if you have completely unstructured data is to disable completely the mapping for the index like this:

curl -X PUT "localhost:9200/my_index?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "enabled": false 
  }
}
'

if you are using python you can do this:

from elasticsearch import Elasticsearch

# Connect to the elastic cluster
es=Elasticsearch([{'host':'localhost','port':9200}])

request_body = {
        "mappings": {
                 "enabled": False
    }
}
es.indices.create(index = 'my_index', body = request_body)

Upvotes: 0

Lordran
Lordran

Reputation: 683

You must know about that the below part just mean that ES could'nt create a type dynamically.

"mapper" : {
        "dynamic" : false
    }

You should configure ES like this:

PUT http://localhost:9200/test_idx/_mapping/test_type
{
  "dynamic":"strict"
}

Then you cant't index other field that without mapping any more ,and get an error as follow:

mapping set to strict, dynamic introduction of [hatae] within [data] is not allowed

If you wanna store the data,but make the field can't be index,you could take the setting like this:

PUT http://localhost:9200/test_idx/_mapping/test_type
{
  "dynamic":false
}

Hope these can help the people with the same issue :).

Upvotes: 5

Andrei Stefan
Andrei Stefan

Reputation: 52368

You're almost there: the value needs to be set to strict. And the correct usage is the following:

PUT /test_idx
{
  "mappings": {
    "test_type": {
      "dynamic":"strict",
      "properties": {
        "field1": {
          "type": "string"
        }
      }
    }
  }
}

And pushing this a bit further, if you want to forbid the creation even of new types, not only fields in that index, use this:

PUT /test_idx
{
  "mappings": {
    "_default_": {
      "dynamic": "strict"
    },
    "test_type": {
      "properties": {
        "field1": {
          "type": "string"
        }
      }
    }
  }
}

Without _default_ template:

PUT /test_idx
{
  "settings": {
    "index.mapper.dynamic": false
  },
  "mappings": {
    "test_type": {
      "dynamic": "strict",
      "properties": {
        "field1": {
          "type": "string"
        }
      }
    }
  }
}

Upvotes: 21

Related Questions