CODEWITHSUNDEEP
elasticsearchlogstash
laymo
laymo

Reputation: 17

How to use mapping in elasticsearch?

After treating logs with logstash, All my fields have the same type 'STRING so i want to use mapping in elasticsearch to change some type like ip, port ect.. whereas i don't know how to do it, i'm a super beginner in ElasticSearch..

Any help ?

Upvotes: 0

Views: 664

Answers (1)

Jakub Kotowski
Jakub Kotowski

Reputation: 7571

The first thing to do would be to install the Marvel plugin in Elasticsearch. It allows you to work with the Elasticsearch REST API very easily - to index documents, modify mappings, etc.

Go to the Elasticsearch folder and run:

bin/plugin -i elasticsearch/marvel/latest

Then go to http://localhost:9200/_plugin/marvel/sense/index.html to access Marvel Sense from which you can send commands. Marvel itself provides you with a dashboard about Elasticsearch indices, performance stats, etc.: http://localhost:9200/_plugin/marvel/

In Sense, you can run:

GET /_cat/indices

to learn what indices exist in your Elasticsearch instance.

Let's say there is an index called logstash.

You can check its mapping by running:

GET /logstash/_mapping

Elasticsearch will return a JSON document that describes the mapping of the index. It could be something like:

{
   "logstash": {
      "mappings": {
         "doc": {
            "properties": {
               "Foo": {
                  "properties": {
                     "x": {
                        "type": "String"
                     },
                     "y": {
                        "type": "String"
                     }
                  }
               }
            }
         }
      }
   }
}

...in this case doc is the document type (collection) in which you index documents. In Sense, you could index a document as follows:

PUT logstash/doc/1
{
  "Foo": {
    "x":"500",
    "y":"200"
  }
}

... that's a command to index the JSON object under the id 1.

Once a document field such as Foo.x has a type String, it cannot be changed to a number. You have to set the mapping first and then reindex.

First delete the index:

DELETE logstash

Then create the index and set the mapping as follows:

PUT logstash
PUT logstash/doc/_mapping
{
   "doc": {
      "properties": {
         "Foo": {
            "properties": {
               "x": {
                  "type": "long"
               },
               "y": {
                  "type": "long"
               }
            }
         }
      }
   }
}

Now, even if you index a doc with the properties as JSON strings, Elastisearch will convert them to numbers:

PUT logstash/doc/1
{
  "Foo": {
    "x":"500",
    "y":"200"
  }
}

Search for the new doc:

GET logstash/_search

Notice that the returned document, in the _source field, looks exactly the way you sent it to Elasticsearch - that's on purpose, Elasticsearch always preserves the original doc this way. The properties are indexed as numbers though. You can run a range query to confirm:

GET logstash/_search
{
  "query":{
    "range" : {
        "Foo.x" : {
            "gte" : 500
        }
    }
  }
}

With respect to Logstash, you might want to set a mapping template for index name logstash-* since Logstash creates new indices automatically: http://www.elastic.co/guide/en/elasticsearch/reference/1.5/indices-templates.html

Upvotes: 2

Related Questions