CODEWITHSUNDEEP
elasticsearchlogstash
Steven Davis
Steven Davis

Reputation: 63

Specifying Field Types Indexing from Logstash to Elasticsearch

I have successfully ingested data using the XML filter plugin from Logstash to Elasticsearch, however all the field types are of the type "text."

Is there a way to manually or automatically specify the correct type?

Upvotes: 1

Views: 1070

Answers (2)

phospodka
phospodka

Reputation: 1028

What you want to do is specify a mapping template. 

PUT _template/template_1
{
  "index_patterns": ["te*", "bar*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z YYYY"
        }
      }
    }
  }
}

Change the settings to match your needs such as listing the properties to map what you want them to map to.

Setting index_patterns is especially important because it tells elastic how to apply this template. You can set an array of index patterns and can use * as appropriate for wildcards. i.e logstash's default is to rotate by date. They will look like logstash-2018.04.23 so your pattern could be logstash-* and any that match the pattern will receive the template.

If you want to match based on some pattern, then you can use dynamic templates.

Edit: Adding a little update here, if you want logstash to apply the template for you, here is a link to the settings you'll want to be aware of.

Upvotes: 1

Tomer
Tomer

Reputation: 559

I found the following technique good for my use:

Logstash would filter the data and change a field from the default - text to whatever form you want. The documentation would be found here. The example given in the documentation is:

filter {
  mutate {
    convert => { "fieldname" => "integer" }
  }
}

This you add in the /etc/logstash/conf.d/02-... file in the body part. I believe the downside of this practice is that from my understanding it is less recommended to alter data entering the ES.

After you do this you will probably get the this problem. If you have this problem and your DB is a test DB that you can erase all old data just DELETE the index until now that there would not be a conflict (for example you have a field that was until now text and now it is received as date there would be a conflict between old and new data). If you can't just erase the old data then read into the answer in the link I linked.

Upvotes: 1

Related Questions