CODEWITHSUNDEEP
elasticsearchmergesegmentthrottling
Sergey O
Sergey O

Reputation: 53

ElasticSearch 6.0.1: .../_forcemerge API does not scale with more threads (is throttled?)

I am using ES 6.0.1 and have to do frequent index open/append/close patterns over many indexes, typically from different clients in parallel. (Yes, I have to open and close every time)

This leads to high number of small Lucene segments per index, and the mentioned sequence becomes slower over time (2-5 times slower sometimes). Default ES segment merging strategy apparently is not doing a great job.

When I use Force Merge API to merge segments within index, the performance of my sequence returns back to normal for processed indices. However, due to large number of indices, I have to apply it many times to handle all indices. Naturally, I run it in multiple threads (connections), but it seems like ES never parallels this operation and the resulting rate of merge is the same, no matter how many parallel requests I make.

I've read and tried things from here but this didn't help.

Could someone kindly suggest any w/a for that?

Upvotes: 1

Views: 441

Answers (1)

Dian Bakti
Dian Bakti

Reputation: 320

You can change size of force_merge thread pool via elasticsearch config file, for example:

thread_pool.force_merge.size: 5

And don't forget to restart Elasticsearch after the config change.

Upvotes: 1

Related Questions