CODEWITHSUNDEEP
logging
Cyclops
Cyclops

Reputation: 689

Best technology or language to manipulate load of log data?

I need to create a solution that needs to get load of data from an elasticsearch db to the program which are applications logs in JSON format. There can be thousands of logs for one request and I need to find the co-relation between then by reading them and finally visualize an output to the user with proper grouping mechanism(kind of mapping in business) in a UI in web browser. Can you guys please suggest me the ideal technology if exist or ideal programming language to accomplished this. I tried to do this grouping with Kibana UI, but the mechanism and the way we want is not there. This can be a inappropriate question but I need a guidance of a experienced people.

Upvotes: 0

Views: 171

Answers (1)

Hugodby
Hugodby

Reputation: 1183

First, have a look at the ElasticSearch documentation about the aggregations : https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html . There are a lot of powerful "grouping mechanism" you can do.

For the ouput, once you have calculated the data you want, you can then push these data again in ElasticSearch and then display them in Kibana. Another solution is to build your own UI.

For the language you can try javascript with NodeJS. It will be easier to manipulate JSON. It also give you the ability to perform asynchronous tasks if your goal is to make your own UI.

Upvotes: 1

Related Questions