CODEWITHSUNDEEP
elasticsearchlogstash
0x4a6f4672
0x4a6f4672

Reputation: 28245

Elasticsseach can’t merge a non object mapping with an object mapping

Somehow it is possible to define a geo-point field type in the Elasticsearch mapping and to import data, but not both. In the JSON data the location fields look like this

"location": { 
  "lat": 41.12,
  "lng": -71.34
}

Because we need "lon" instead of "lng" we use this "mutate" filter in the Logstash configuration to rename the field:

mutate {
    rename => {
        "[location][lng]" => "[location][lon]"
    }
}

If we do not use a mapping then Elasticsearch uses the following mapping automatically for location fields and imports the data

"location": {
  "properties": {
    "lat": {
      "type": "float"
    },
    "lon": {
      "type": "float"
    }
  }
}

So far so good. But if I use “geo_point” now in the Elasticsearch mapping when I create the index then I can not import any data anymore because I get the error message “can’t merge a non object mapping [location] with an object mapping” in Logstash which can happen if we try to change a mapping. But here the mapping was already used to create the index:

"mappings":{
    "properties":{
        "location": {
            "type": "geo_point",
            "ignore_malformed": "true",
        },
    }
}

Apparently Logstash and Elasticsearch consider the field location which has the type geo_point in the mapping as something which is not an object, while the JSON data for this location is an object.

While it is not possible to import the data in Logstash using this mapping, I can save the document in the Kibana DEV Tools like this though

PUT geo-test/_doc/1
{
  "title": "Geo-point test",
  "location": { 
    "lat": 41.12,
    "lon": -71.34
  }
}

How is it possible to import the data in Logstash using a geo-point mapping? (I am using Elasticsearch version 7.9.1 and Logstash version 7.12.0 including S3 Input plugin and Elasticsearch Output plugin)

Upvotes: 1

Views: 3814

Answers (1)

Val
Val

Reputation: 217274

The reason you get this error is because you've specified a different document_type in your elasticsearch output. Simply remove the emphasized line below and it will work the way you expect:

elasticsearch {
    hosts => [ "${ES_DOMAIN_NAME}:443" ]
    healthcheck_path => "/"
    ilm_enabled => false
    ssl => true
    index => "${ES_INDEX}"
    document_type => "json"         <--- remove this line
}

The mapping you've installed is meant for the default document type which is _doc not json.

Upvotes: 2

Related Questions