CODEWITHSUNDEEP
elasticsearchaws-elasticsearch
focode
focode

Reputation: 716

Reindex on elastic search taking long time

This is the scenario: In Index1 I have more than 60 00000 records, In the same index I am putting data at the rate of 630 records per seconds and then I created another index Index2 and initiated reindex as per this document : https://www.elastic.co/guide/en/elasticsearch/reference/6.8/docs-reindex.html

It took more than 40 minutes to transfer 1000 records in a new index, but If I am not doing put at all then the same reindex takes 5 minutes.

Is there a way I can reindex at a faster rate even when I am putting data at 600 records per seconds and why It is taking so long time I am using AWS elastic search version 7.7, m4.large.elasticsearch 2 nodes

Upvotes: 1

Views: 6776

Answers (3)

Amit
Amit

Reputation: 32376

In Addition to @Ibexit and @Jay answers, wanted to highlight that there could be multiple cause of slowness which could be due to ES settings and the application which you are using(and the way you are using) to re-index the data.

I have written a blog which covers both ES and application side issues and highlight the short tips to improve specifically re-indexing performance.

Upvotes: 1

Jaycreation
Jaycreation

Reputation: 2089

With this indexing slowness, ibexit is probably right. You have an issue with your HDD.

But their is also some best practices that can save you some time. First advice will be to disable replicas and refresh on you destination index for indexing time. And re-enable it after completion. index.number_of_replicas: 0 index.refresh_interval: -1

You can read a very good article here: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html

Upvotes: 3

ibexit
ibexit

Reputation: 3667

Elastic´s read, and especially write speed, is mainly limited by I/O constraints of the underlying storage. If you stop indexing, the reindex will speed up because there are more IOPS and bandwidth available for reindexing.

Try a faster disk with SSD or NVMe-SSD and prevent network-attached (ESB) storage in favor of instance storage volumes, at least provision the required IOPS + Bandwidth in the chosen Storage.

Upvotes: 1

Related Questions