CODEWITHSUNDEEP
elasticsearchdate-formattingelasticsearch-mapping
Mark B
Mark B

Reputation: 4928

How can I apply dynamic date formats to multiple types in Elasticsearch?

I'm trying to get Elasticsearch to recognise strings in the format yyyy-MM-dd HH:mm:ss as date fields. I've created a dynamic date format and applied that to the default mapping. It works nicely when I index documents of my first type - any new fields where the data is in this format get initialised as date fields.

The problem comes when I try to create documents of a new type, but with date format fields with the same name as in my first type. These fail with a malformed date error.

Here's an example set of Kibana commands to demonstrate:

DELETE /datetest

PUT /datetest

PUT /datetest/_mapping/_default_
{
  "dynamic_date_formats" : ["yyyy-MM-dd HH:mm:ss"]
}

PUT /datetest/doc/1
{
  "date" : "2015-01-01 12:00:00"
}

# This one works fine
PUT /datetest/otherdoc/1
{
  "otherdate" : "2015-01-01 12:00:00"
}

# This one does not
PUT /datetest/otherdoc/2
{
  "date" : "2015-01-01 12:00:00"
}

The last command gives this error: "Invalid format: \"2015-01-01 12:00:00\" is malformed at \" 12:00:00\""

I know that fields with the same name in different types must have the same data type, but in this case, I want them to have the same data type - date. I could manually create the mappings for each new type, but I want it to automatically support new types added to my source data. It seems to be what the dynamic date format is supposed to do. Am I doing something wrong here?

Upvotes: 5

Views: 5016

Answers (2)

Matthew A Thomas
Matthew A Thomas

Reputation: 914

I would create a custom dynamic template. Something like this:

PUT /datetest/_mapping/_default_
{
    "date_detection" : true,
    "dynamic_templates" : [
        {
            "dates" : {
               "match" : ".*date.*",
                "mapping" : {
                    "type" : "date",
                    "format" : 'yyyy-MM-dd HH:mm:ss'
                }
            }
        }
    ]
}

Just tried it. It seems to work. I hope this helps :)

Upvotes: 2

Evaldas Buinauskas
Evaldas Buinauskas

Reputation: 14077

All I can think of, would be adding dynamic template:

PUT /datetest
{
  "mappings": {
    "_default_": {
      "date_detection": false,
      "dynamic_templates": [
        {
          "dates": {
            "match": ".*Date|date",
            "match_pattern": "regex",
            "mapping": {
              "type": "date",
              "format": "yyyy-MM-dd HH:mm:ss||dateOptionalTime"
            }
          }
        }
      ]
    }
  }
}

All three statements will run just fine:

PUT /datetest/doc/1
{ "date" : "2015-01-01 12:00:00" }
PUT /datetest/otherdoc/1
{ "otherdate" : "2015-01-01 12:00:00" }
PUT /datetest/otherdoc/2
{ "date" : "2015-01-01 12:00:00" }

Dynamic template will try to match your field name on based pattern. They need to end with date or Date.

I know you need just yyyy-MM-dd HH:mm:ss format, but I've also added default one, so that Elasticsearch can pick one from multiple ones.

Upvotes: 1

Related Questions