How to search on a URL exactly in ElasticSearch / Kibana

Question

I have imported an IIS log file and the data has moved through Logstash (1.4.2), into ElasticSearch (1.3.1) and then being displayed in Kibana.

My filter section is as follows:

filter {
  grok {
     match => 
        ["message" , "%{TIMESTAMP_ISO8601:iisTimestamp} %{IP:serverIP} %{WORD:method} %{URIPATH:uri} - %{NUMBER:port} - %{IP:clientIP} - %{NUMBER:status} %{NUMBER:subStatus} %{NUMBER:win32Status} %{NUMBER:timeTaken}"]
  }
}

When using a Terms panel in Kibana, and using "uri" (one of my captured fields from Logstash), it is matching the tokens within the URI. Therefore it is matching items like:

• 'Scripts'
• '/'
• 'EN

Q: How do I display the 'Top URLs' in their full form?

Q: How do I inform ElasticSearch that the field is 'not_analysed'. I don't mind having 2 fields, for example:

• uri - The tokenized URI
• uri.raw - the fully formed URL.

Can this be done Logstash side, or is this a mapping that needs to be set up in ElasticSearch?

---

Mapping is as follows :

//http://localhost:9200/iislog-2014.10.09/_mapping?pretty

{
  "iislog-2014.10.09" : {
    "mappings" : {
      "iislogs" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "@version" : {
            "type" : "string"
          },
          "clientIP" : {
            "type" : "string"
          },
          "device" : {
            "type" : "string"
          },
          "host" : {
            "type" : "string"
          },
          "id" : {
            "type" : "string"
          },
          "iisTimestamp" : {
            "type" : "string"
          },
          "logFilePath" : {
            "type" : "string"
          },
          "message" : {
            "type" : "string"
          },
          "method" : {
            "type" : "string"
          },
          "name" : {
            "type" : "string"
          },
          "os" : {
            "type" : "string"
          },
          "os_name" : {
            "type" : "string"
          },
          "port" : {
            "type" : "string"
          },
          "serverIP" : {
            "type" : "string"
          },
          "status" : {
            "type" : "string"
          },
          "subStatus" : {
            "type" : "string"
          },
          "tags" : {
            "type" : "string"
          },
          "timeTaken" : {
            "type" : "string"
          },
          "type" : {
            "type" : "string"
          },
          "uri" : {
            "type" : "string"
          },
          "win32Status" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

Answer 1

The problem is that the iislog- is not compliant with the logstash- format, and hence did not pick up the template:

My index format was iislog-YYYY.MM.dd, this did not use the out-of-the-box mappings by Logstash. When using the logstash- index format, Logstash will create 2 pairs of fields for strings. For example uri is:

• uri (appears in Kibana)
• uri.raw (does not appear in Kibana)

Note that the uri.raw will not appear in Kibana - but it is queryable.

So the solution to use an alternative index is to:

1. Don't bother! Use the default index format of logstash-%{+YYYY.MM.dd}

2. Add a "type" to the file input to help you filter the correct logs in Kibana (whilst using the logstash- index format)

   input { 
     file {
         type => "iislog"
         ....
     }
   }
   

3. Apply filtering in Kibana based in the type

OR

If you really really do want a different index format:

1. Copy the default configuration file to a new file, say iislog-template.json

2. Reference the configuration file in the output ==> elasticsearch like this:

   output {    
      elasticsearch_http {
         host => localhost
         template_name => "iislog-template.json"
         template => "<path to template>"
         index => "iislog-%{+YYYY.MM.dd}"   
      }
   }
   

Answer 2

In your Elasticsearch mapping:

url: {
  type: "string",
  index: "not_analyzed"
}