CODEWITHSUNDEEP
elasticsearch
kiran
kiran

Reputation: 497

elasticsearch: is creating one index for each log good?

I am using elasticsearch to index logs from an automation run of test cases. I am creating an index for each of the runs (that can have from 1000 to million events). I create about 200 indices per day. Is this a good methodology to create an index for each run or should I just have 1 index and then put all the logs from multiple runs into this index? The amount of data is huge and so I chose separate indices. I am expecting 200 logs everyday each with 1million events. Please help me

Upvotes: 0

Views: 103

Answers (1)

Jilles van Gurp
Jilles van Gurp

Reputation: 8294

Depends how long you want to retain your data and the size of your cluster. At 200 indices per day, each with lots of associated files, you're looking at a lot of file handles. So, that doesn't sound like it would scale beyond a few weeks or months on a very small cluster since you'll be running out of file handles.

A better strategy might be to do what logstash does by default which is to create a new index every day. Then your next choice will be to play with the number of shards and nodes in the cluster. Assuming you want to store a worst case of 200M log entries per day on a 3 or 5 node cluster, probably the default of 5 shards is fine. If you go for more nodes, you'll probably want more shards so that each shard is smaller. Also consider using elasticsearch curator to e.g. close older indices and optimize them.

Upvotes: 1

Related Questions