CODEWITHSUNDEEP
elasticsearchlogstashkibana
Bmis13
Bmis13

Reputation: 670

Distributed Tracing logging and Integrating with Logstash,Kibana and ElasticSearch

I have implemented a Distributed Transaction Logging library with Tree like Structure as mention in Google Dapper(http://research.google.com/pubs/pub36356.html) and eBay CAL Transaction Logging Framework(http://devopsdotcom.files.wordpress.com/2012/11/screen-shot-2012-11-11-at-10-06-39-am.png).

Log Format

TIMESTAMP   HOSTNAME  DATACENTER  ENVIRONMENT  EVENT_GUID PARENT_GUID TRACE_GUID APPLICATION_ID TREE_LEVEL TRANSACTION_TYPE TRANSACTION_NAME STATUS_CODE DURATION(in ms) PAYLOAD(key1=value2,key2=value2)

GUID HEX NUMBER FORMAT

MURMER_HASH(HOSTNAME + DATACENTER + ENVIRONMENT)-JVM_THREAD_ID-(TIME_STAMP+Atomic Counter)

What I would like to do is to integrate this format with Kibana UI and when user want to search and click on on TRACE_GUID it will show something similar to Distributed CALL graph which show where the time was spent. Here is UI http://twitter.github.io/zipkin/. This will be great. I am not UI developer if some can point me how to do this that will be great.

Also I would like to know how I can index elastic search payload data so user specify some expression like in payload (duration > 1000) then, Elastic Search will bring all the loglines that satisfy condition. Also, I would like to index Payload as Name=Value pair so user can query (key3=value2 or key4 = exception) some sort of regular expression. Please let me know if this can be achieved. Any help pointer would be great..

Thanks, Bhavesh

Upvotes: 4

Views: 3007

Answers (1)

Alain Collins
Alain Collins

Reputation: 16362

The first step to good searching in elasticsearch is to create fields from your data. With logs, logstash is the proper tool. The grok{} filter uses patterns (existing or user-defined regexps) to split the input into fields.

You would need to make sure that it was mapped to an integer (e.g. %{INT:duration:int} in your pattern). You could then query elasticsearch for "duration:>1000" to get the results.

Elasticsearch uses the lucene query engine, so you can find sample queries based on that.

Upvotes: 1

Related Questions