CODEWITHSUNDEEP
loggingelasticsearchlogstashkibana
Vagif Abilov
Vagif Abilov

Reputation: 9991

How to aggregate in Kibana information from multiple Elasticsearch indexes?

We are setting up logs from several related applications so the log events are imported into Elasticsearch (via Logstash). It was straightforward create Kibana dashboards to visualize log indexes for each application, but since the applications are related and its activities belong to the same pipeline, it would be great to build a dashboard that would show aggregated information, collected from different applications. Such dashboard would be especially useful to track failures and performance problems.

Right now I can see three main ways to implement aggregated dashboard:

  1. Keep separate application logs and configure Kibana dashboard that would consume information from different applications. I am afraid this can be a challenging task, I am not even sure Kibana fully supports it.
  2. Revise application logging so they will all log to the same index. What I dislike about this is that log event structure must be then unified across applications, and they are built by different people in different languages. I've lost my faith to centralized control over such low level details like logging.
  3. Keep applications log and corresponding Elastichsearch indexes as they are now, but set up a new index which will contain aggregate information. This article describes how to configure Elasticsearch to dump it’s logs to Logstash which would then insert them back into Elasticsearch for searching. At first glance this approach may look surprising: why would you need to re-insert log data once again into the same database? It's another index, it adds overhead, uses more space etc. But it gives the opportunity to set up the index in a way that will be suitable for a aggregated Kibana dashboard.

I wonder if someone has gone through a similar dilemma and can share their experience.

Upvotes: 3

Views: 9507

Answers (2)

antonbormotov
antonbormotov

Reputation: 1987

We faced the same problem, but in different perspective.

I needed to get data from 2 indices in Kibana. Our data structure is the same in both indices.

So, I added second index manually (section Settings->Objects):

{
  "index": [
    "index_one",
    "index_two"
  ],
 ...
}

It helped me to get data from those indices that I need.

Upvotes: 1

bpeirce
bpeirce

Reputation: 100

I believe you can just set the Default Index to _all if you're not planning to use timestamped indices.

Using menus, go to Configure, click the Index tab, and set Timestamping to "none" and Default Index to _all. The JSON schema would end up containing something like this:

  "index": {
    "interval": "none",
    "pattern": "[logstash-]YYYY.MM.DD",
    "default": "_all",
    "warm_fields": false
  },

If you need timestamped indices, you would need to choose the approriate interval and enter a comma-separated list of the indices, each specified in the proper format.

Upvotes: 3

Related Questions