CODEWITHSUNDEEP
searchelasticsearchkibana
A_Elric
A_Elric

Reputation: 3568

Kibana: Search within text for string

I have A log message in Kibana that contains this:

org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:597)

Actual search that isn't returning results: log_message: "hibernate3"

If I search for "hibernate3" this message will not appear. I am using an Elasticsearch template and have indexed the field, but also want to be able to do case-insensitive full-text searching. Is this possible?

Template that is in use:

{
"template": "filebeat-*",
"mappings": {
    "mainProgram": {
        "properties": {
            "@timestamp": {
                "type": "date",
                "format": "strict_date_optional_time||epoch_millis"
            },
            "@version": {
                "type": "text"
            },
            "beat": {
                "properties": {
                    "hostname": {
                        "type": "text"
                    },
                    "name": {
                        "type": "text"
                    }
                }
            },
            "class_method": {
                "type": "text",
                "fielddata": "true",
                "index": "true"
            },
            "class_name": {
                "type": "text",
                "fielddata": "true"
            },
            "clientip": {
                "type": "ip",
                "index": "not_analyzed"
            },
            "count": {
                "type": "long"
            },
            "host": {
                "type": "text",
                "index": "not_analyzed"
            },
            "input_type": {
                "type": "text",
                "index": "not_analyzed"
            },
            "log_level": {
                "type": "text",
                "fielddata": "true",
                "index": "true"
            },
            "log_message": {
                "type": "text",
                "index": "true"
            },
            "log_timestamp": {
                "type": "text"
            },
            "log_ts": {
                "type": "long",
                "index": "not_analyzed"
            },
            "message": {
                "type": "text"
            },
            "offset": {
                "type": "long",
                "index": "not_analyzed"
            },
            "query_params": {
                "type": "text",
                "index": "true"
            },
            "sessionid": {
                "type": "text",
                "index": "true"
            },
            "source": {
                "type": "text",
                "index": "not_analyzed"
            },
            "tags": {
                "type": "text"
            },
            "thread": {
                "type": "text",
                "index": "true"
            },
            "type": {
                "type": "text"
            },
            "user_account_combo": {
                "type": "text",
                "index": "true"
            },
            "version": {
                "type": "text"
            }
        }
    },
    "access": {
        "properties": {
            "@timestamp": {
                "type": "date",
                "format": "strict_date_optional_time||epoch_millis"
            },
            "@version": {
                "type": "text"
            },
            "beat": {
                "properties": {
                    "hostname": {
                        "type": "text"
                    },
                    "name": {
                        "type": "text"
                    }
                }
            },
            "clientip": {
                "type": "ip",
                "index": "not_analyzed"
            },
            "count": {
                "type": "long",
                "index": "not_analyzed"
            },
            "host": {
                "type": "text",
                "index": "true"
            },
            "input_type": {
                "type": "text",
                "index": "not_analyzed"
            },
            "log_timestamp": {
                "type": "text"
            },
            "log_ts": {
                "type": "long",
                "index": "not_analyzed"
            },
            "message": {
                "type": "text"
            },
            "offset": {
                "type": "long",
                "index": "not_analyzed"
            },
            "query_params": {
                "type": "text",
                "index": "true"
            },
            "response_time": {
                "type": "long"
            },
            "sessionid": {
                "type": "text",
                "index": "true"
            },
            "source": {
                "type": "text",
                "index": "not_analyzed"
            },
            "statuscode": {
                "type": "long"
            },
            "tags": {
                "type": "text"
            },
            "thread": {
                "type": "text",
                "index": "true"
            },
            "type": {
                "type": "text",
                "index": "true"
            },
            "uripath": {
                "type": "text",
                "index": "true"
            },
            "user_account_combo": {
                "type": "text",
                "index": "true"
            },
            "verb": {
                "type": "text",
                "index": "true"
            }
        }
    }
}
}

Upvotes: 31

Views: 161634

Answers (4)

Doctor
Doctor

Reputation: 7966

For me it was because I was using the ".keyword".

My key was called "message" and I had "message" and "message.keyword" available.

Full text search isn't working on ".keyword".

Not working :

message.keyword : hello

Working :

message : hello

Upvotes: 5

Olivier Tonglet
Olivier Tonglet

Reputation: 3502

With Kibana 6.4.1 I used the "%" as wildcard.

message: %hibernate3%

Upvotes: 12

Kulasangar
Kulasangar

Reputation: 9434

According to your scenario, what you're looking for is an analyzed type string which would first analyze the string and then index it. A quote from the doc.

In other words, index this field as full text.

Thus make sure that, you have your mapping of the necessary fields properly so that you'll be able to do a full-text search on the docs.

Assuming that, in Kibana if the log line is under the field message, you could simply search for the word by:

message:"hibernate3"

You might also want to refer this, to identify the variance between Term Based and Full-Text.

EDIT

Have the mapping of the field log_message as such:

"log_message": {
       "type": "string", <- to make it analyzed
       "index": "true"
}

Also try doing a wildcard search as such:

{"wildcard":{"log_message":"*.hibernate3.*"}}

Upvotes: 15

Eljah
Eljah

Reputation: 5155

message: *.hibernate3.*

also works (please note, that no quotes are needed for that)

Upvotes: 32

Related Questions