CODEWITHSUNDEEP
elasticsearchlogstash
user4369887
user4369887

Reputation: 51

can i set logstash default elasticsearch mapping through elasticsearch-template.json

I use logstash + elasticsearch to collect syslog and want to set ttl for log ageing

I find a file named elasticsearch-template.json in the logstash,the path is logstash/logstash-1.4.2/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

I add ttl info in the file like this:

{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
        "_ttl": {
         "enabled": true,
         "default": "1d"
       },
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "path": "full",
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}

then restart logstash, delete all elasticsearch index. I check the new index's mapping in the elasticsearch, but it didn't work in this way.

How can I config the index template?

Upvotes: 5

Views: 16533

Answers (3)

Taras Melnyk
Taras Melnyk

Reputation: 3265

I've created new template.json file and defined path to it into elasticsearch output block of logstash.yml config file:

stdout { codec => json_lines }

elasticsearch {

"hosts" => ["ip:port"]
"index" => "name-of-index-%{+dd.MM.YYYY}"
template => "/{path-to-logstash-folder}/templates/your-template.json"
template_overwrite => true
manage_template => false
}

document_type for Elastic I defined into input block of logstash.yml config file:

input {

file {
path => "/your-path-to-directory/*.log"
type => "name-of-type"
 }
}

There is my template.json file

{
"name-of-index": {
"order": 0,
"version": 50001,
"template": "name-of-index-*",
"settings": {
  "index": {
    "refresh_interval": "5s"
  }
},
"mappings": {
  "_default_": {
    "dynamic_templates": [
      {
        "message_field": {
          "path_match": "message",
          "mapping": {
            "norms": false,
            "type": "text"
          },
          "match_mapping_type": "string"
        }
      },
      {
        "string_fields": {
          "mapping": {
            "norms": false,
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "match_mapping_type": "string",
          "match": "*"
        }
      }
    ],
    "_all": {
      "norms": false,
      "enabled": true
    },
    "properties": {
      "@timestamp": {
        "include_in_all": false,
        "type": "date"
      },
      "geoip": {
        "dynamic": true,
        "properties": {
          "ip": {
            "type": "ip"
          },
          "latitude": {
            "type": "half_float"
          },
          "location": {
            "type": "geo_point"
          },
          "longitude": {
            "type": "half_float"
          }
        }
      },
      "@version": {
        "include_in_all": false,
        "type": "keyword"
      }
    }
  }
},
"aliases": {}
 }
 }

Upvotes: 0

lingxiao
lingxiao

Reputation: 1224

you need to change your logstash configuration.

if you have followed the default settings, logstash has already created a template inside elasticsearch named logstash, logstash will keep on using that template stored in elasticsearch unless you tell it not to explicitly.

modify that template file you found but in addition to that, in your logstash configuration, set the following:

output {
  elasticsearch {
    ...
    template_overwrite => true
    ...
  }
}

Upvotes: 8

tony v
tony v

Reputation: 28

Doesn't look like that JSON file is in the correct folder. Here is the documentation on how to use the templates: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html About the folder:

Config

Index templates can also be placed within the config location (path.conf) under the templates directory (note, make sure to place them on all master eligible nodes). For example, a file called template_1.json can be placed under config/templates and it will be added if it matches an index. Here is a sample of the mentioned file:

Upvotes: 2

Related Questions