CODEWITHSUNDEEP
elasticsearch
JesusTheHun
JesusTheHun

Reputation: 1237

Partially dynamic mapping

In my index "process" I have a type "logs". For this type, I have this mapping:

{
    "process": {
        "mappings": {
            "logs": {
                "properties": {
                    "channel": {
                        "type": "string"
                    },
                    "context": {
                        "type": "object"
                    },
                    "datetime": {
                        "type": "date",
                        "format": "dateOptionalTime"
                    },
                    "extra": {
                        "type": "object"
                    "level": {
                        "type": "long"
                    },
                    "level_name": {
                        "type": "string"
                    },
                    "message": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

The thing is, the "context" property and the "extra" property are dynamic fields, each document can have a different number of extra & context sub elements with different names. When I set this mapping, and then I check http://localhost:9200/process/_mapping/logs, I see this very mapping. So it's worked. Also the answer of the setting is {acknowledge: true} . So far so good.

Then, I add a several documents, or at least I try. The first document changes ES mapping!

The new mapping is:

{
  "process" : {
    "mappings" : {
      "logs" : {
        "properties" : {
          "channel" : {
            "type" : "string"
          },
          "context" : {
            "properties" : {
              "columns" : {
                "type" : "string"
              },
              "count" : {
                "type" : "long"
              },
              "errorMessage" : {
                "type" : "string"
              },
              "serviice" : {
                "type" : "string"
              }
            }
          },
          "datetime" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "extra" : {
            "properties" : {
              "uid" : {
                "type" : "string"
              }
            }
          },
          "level" : {
            "type" : "long"
          },
          "level_name" : {
            "type" : "string"
          },
          "message" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

And when I try to add a new document, it doesn't match this new mapping and trigger this error :

MapperParsingException[object mapping [context] trying to serialize a value with no field associated with it, current value [pid]]

So I am clearly missing something.

  1. Why my mapping get overwritten ? I tried to remove the entire index and to build everything up from scratch, same behavior.
  2. Is what I want to achieve possible ? or do I have the wrong approach ?

EDIT : the document that is inserted and seems to update the mapping :

{
  "message": "Starting background process `{service}`, registered under the ID ",
  "context": {
    "id": null,
    "serviice": "fiduceo.import"
  },
  "level": 250,
  "level_name": "NOTICE",
  "channel": "process",
  "datetime": "2016-06-09T11:23:42.304859+02:00",
  "extra": {
    "uid": "d1cb925"
  }
}

And the one who fails:

{
  "message": "Background process created (pid: {pid})",
  "context": [
    "pid",
    18871
  ],
  "level": 250,
  "level_name": "NOTICE",
  "channel": "process",
  "datetime": "2016-06-09T11:23:41.519456+02:00",
  "extra": {
    "uid": "09fe183"
  }
}

Upvotes: 1

Views: 81

Answers (1)

Haphil
Haphil

Reputation: 1250

Your context field contains an invalid JSON

it should look like this:

"context": {
    "pid": 18871
  },

compared to your version:

"context": [
    "pid",
    18871
  ]

EDIT: In order to give a full answer (and repeat what was written in the comment):

  1. Your mapping is not overwritten, but updated. This happens for the fields that are contained in the new indexed JSON document but not the initial mapping.
  2. It's possible. And your general approach is correct.

Upvotes: 1

Related Questions