elasticsearch dynamic field nested detection

Question

Hi im trying to create an index in my elastic search without defining the mapping so what i did was this.

PUT my_index1/my_type/1
{
  "group" : "fans",
  "user" : [
    {
      "first" : "John",
      "last" :  "Smith",
      "age" :  "1",
      "enabled": false
    },
    {
      "first" : "Alice",
      "last" :  "White",
      "age" :  "10",
      "enabled": true
    }
  ]
}

if did this elastic search will create a mapping for this index which is the result is

{
   "my_index1": {
      "mappings": {
         "my_type": {
            "properties": {
               "group": {
                  "type": "text",
                  "fields": {
                     "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                     }
                  }
               },
               "user": {
                  "properties": {
                     "age": {
                        "type": "text",
                        "fields": {
                           "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                           }
                        }
                     },
                     "enabled": {
                        "type": "boolean"
                     },
                     "first": {
                        "type": "text",
                        "fields": {
                           "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                           }
                        }
                     },
                     "last": {
                        "type": "text",
                        "fields": {
                           "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                           }
                        }
                     }
                  }
               }
            }
         }
      }
   }
}

if you would notice the property user didn't have a type of nested other properties has their own type defined by elastic search is there a way to it automatically the mapping should be look like this for the user property

"user": {
type:"nested"
                  "properties": {
                     "age": {
                        "type": "text",
                        "fields": {
                           "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                           }
                        }
                     },
                     "enabled": {
                        "type": "boolean"
                     },
                     "first": {
                        "type": "text",
                        "fields": {
                           "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                           }
                        }
                     },
                     "last": {
                        "type": "text",
                        "fields": {
                           "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                           }
                        }
                     }
                  }
               }
            }

which is missing. im currently using nest

is there a way to define a dynamic mapping to detect if the newly added data on index is nested?

Answer 1

By default, Elasticsearch/Lucene has no concept of inner objects. Therefore, it flattens object hierarchies into a simple list of field names and values.

The above document would be converted internally into a document that looks more like this: (See Nested field type for more details)

{
  "group" :        "fans",
  "user.first" : [ "alice", "john" ],
  "user.last" :  [ "smith", "white" ]
}

There is no beautiful answer here. A common approach might be using dynamic template to convert object to nested (however, a side effect is that all fields of object type would be changed to nested type),

{
    "mappings": {
        "dynamic_templates": [
            {
                "objects": {
                    "match": "*",
                    "match_mapping_type": "object",
                    "mapping": {
                        "type": "nested"
                    }
                }
            }
        ]
    }
}

Another approach is specify mapping for the field before inserting data.

PUT <your index>
{
  "mappings": {
    "properties": {
      "user": {
        "type": "nested" 
      }
    }
  }
}

Answer 2

You can define a dynamic template where you can define your own custom mapping which can be used later when you index documents in the index.

Adding a step by step procedure, with the help of which automatically the mapping of the user field would be mapped to that of nested type

First, you need to define a dynamic template for the index as shown below, which have a match parameter which will match the field name having pattern similar to user* and map it to nested type

PUT /<index-name>
{
  "mappings": {
    "dynamic_templates": [
      {
        "nested_users": {
          "match": "user*",
          "mapping": {
            "type": "nested"
          }
        }
      }
    ]
  }
}

After creating this template, you need to index the documents into it

POST /<index-name>/_doc/1
{
  "group": "fans",
  "user": [
    {
      "first": "John",
      "last": "Smith",
      "age": "1",
      "enabled": false
    },
    {
      "first": "Alice",
      "last": "White",
      "age": "10",
      "enabled": true
    }
  ]
}

Now when you see the mapping of the index documents, using the Get Mapping API, the mapping would be similar to what you expect to see

GET /<index-name>/_mapping?pretty
{
  "index-name": {
    "mappings": {
      "dynamic_templates": [
        {
          "nested_users": {
            "match": "user*",
            "mapping": {
              "type": "nested"
            }
          }
        }
      ],
      "properties": {
        "group": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "user": {
          "type": "nested",                  // note this
          "properties": {
            "age": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "enabled": {
              "type": "boolean"
            },
            "first": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "last": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }
  }
}

Or as @Jacky1205 mentioned, if it is not field-specific then you can use the below template that will match all object type fields to be of nested type

{
  "mappings": {
    "dynamic_templates": [
      {
        "nested_users": {
          "match": "*",
          "match_mapping_type": "object",
          "mapping": {
            "type": "nested"
          }
        }
      }
    ]
  }
}