CODEWITHSUNDEEP
elasticsearchschemalogstashkibana
GoatInTheMachine
GoatInTheMachine

Reputation: 3793

ElasticSearch - mismatching field types in schema

I've got an ELK stack running with Logstash pulling log events from RabbitMQ. There are a number of loggers that are writing to the queue, and although they all write JSON with a similar schema, one of them sends across a particular field as an Int, while the others send it across as a String.

If the first message to get indexed contains a string, everything works fine, but it seems that if the first message contains an int, then the field type in the index is then Int, and documents where it is a string fail to index.

Is there a way to define the schema ES should use ahead of time? Or is the best solution to change the Int logger to send the number across as a String?

Upvotes: 1

Views: 443

Answers (1)

Daniel Bubenheim
Daniel Bubenheim

Reputation: 4229

you should define a mapping for this particular field, otherwise elasticsearch "guesses" it from the first ocurrence.

"name": {
    "type": "string"
}

see also: https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html and https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html

When you index a document that contains a new field—one previously not seen—Elasticsearch will use dynamic mapping to try to guess the field type from the basic datatypes available in JSON

Upvotes: 3

Related Questions