CODEWITHSUNDEEP
performanceelasticsearch
David542
David542

Reputation: 110267

Slow aggregation queries while ingesting data

I have a large aggregation query that comes incredibly slow when I am updating my data. I am not saving the data to a tmp index (and then renaming it when it's done) but saving it directly to the index I'm querying.

Upvotes: 0

Views: 60

Answers (1)

Egor
Egor

Reputation: 271

It's hard to tell without any details, as there can be many factors affecting performance.

In general, though, indexing is a computationally intensive operation, so while it may feel counterintuitive, but as well as looking at how to improve your search, I'd have a look at how you can optimize your indexing to reduce load it causes.

In my experience, I have had a somewhat similar problem. What I observed was high IO utilization, while indexing progress coming to a halt and search pretty much not available. And I had good results with tuning configuration related to segments and merging, which can have a pretty bad effect on spinning disks as an index grows and it starts merging big segments. Also, if you don't have strict requirements for new documents availability, changing index.refresh_interval and batching documents for indexing can help a lot.

Have a look at docs here https://www.elastic.co/guide/en/elasticsearch/guide/2.x/indexing-performance.html

Upvotes: 1

Related Questions