CODEWITHSUNDEEP
elasticsearchlogstashelastic-stack
steve
steve

Reputation: 171

elasticsearch es_rejected_execution_exception

I'm trying to index a 12mb log file which has 50,000 logs. After Indexing around 30,000 logs, I'm getting the following error

[2018-04-17T05:52:48,254][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@560f63a9 on EsThreadPoolExecutor[name = EC2AMAZ-1763048/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@7d6ae98b[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 3834]]"})

However, I've gone through the documentation and elasticsearch forum which suggested me to increase the elasticsearch bulk queue size. I tried using curl but I'm not able to do that.

curl -XPUT localhost:9200/_cluster/settings -d '{"persistent" : {"threadpool.bulk.queue_size" : 100}}'

is increasing the queue size good option? I can't increase the hardware because I have fewer data.

The error I'm facing is due to the problem with the queue size or something else? If with queue size How to update the queue size in elasticsearch.yml and do I need to restart es after updating in elasticsearch.yml?

Please let me know. Thanks for your time

Upvotes: 11

Views: 26998

Answers (4)

Cory Robinson
Cory Robinson

Reputation: 5282

The rejected_execution_exception error occurs when the OpenSearch cluster is overwhelmed with requests and cannot process new ones. This can happen when you're bulk indexing a large number of documents. To prevent this error, you can try the following:

Adjust the bulk size: Decrease the number of documents indexed in a single bulk operation to reduce the pressure on your OpenSearch cluster. You can either change the batch size in the osClient.helpers.bulk method or implement a custom function to break your dataset into smaller chunks before indexing, as demonstrated in the previous answer.

Throttle the requests: Add a delay between bulk requests to give the OpenSearch cluster some breathing room. You can modify the wait parameter in the osClient.helpers.bulk method to increase the time interval between bulk requests. For example, you can change wait: 3000 to wait: 5000 or an even higher value.

Increase the number of retries: Increase the number of retries in case of a failure by modifying the retries parameter in the osClient.helpers.bulk method. This will give the operation more chances to succeed when the cluster is under pressure.

Monitor your OpenSearch cluster's performance: Keep an eye on the performance of your OpenSearch cluster, and consider upgrading the cluster's resources (CPU, memory, storage) or the number of nodes to better handle the indexing workload.

Optimize your OpenSearch cluster's settings: You can optimize your OpenSearch cluster's settings, such as adjusting the refresh_interval or the number_of_replicas, to improve indexing performance. Be cautious when making these changes, as they may affect the cluster's stability and query performance.

By applying these recommendations, you should be able to mitigate the rejected_execution_exception error and successfully bulk import your data from MongoDB to OpenSearch. Remember to monitor your cluster's performance and adjust the settings as needed to find the optimal configuration for your specific use case.

Upvotes: 0

Marc
Marc

Reputation: 14391

When moving/indexing large volumes of data it can help to tell ElasticSearch not to try index them immediately. By default it indexes every 3s but during a migration this can cause errors (as you have posted) and also performance problems. You could try setting the refresh_interval to some large value.

POST /myindex/_settings
{
  "settings": {
    "index": {
      "refresh_interval": "600s"
    }
  }
}

After indexing/migration you should set the value back to 3s.

Upvotes: 2

Kumar S
Kumar S

Reputation: 387

by reducing the batch size it resolved my problem.

POST _reindex
{
"source":{
"index":"sourceIndex",
"size": 100
},
"dest":{
"index":"destIndex"}
}

Upvotes: 0

Nirmal
Nirmal

Reputation: 1336

Once your indexing cant keep up with indexing requests - elasticsearch enqueues them in threadpool.bulk.queue and starts rejecting if the # of requests in queue exceeds threadpool.bulk.queue_size

Its good idea to consider throttling your indexing . Threadpool size defaults are generally good ; While you can increase them , you may not have enough resources ( memory, CPU ) available .

This blogpost from elastic.co explains the problem really well .

Upvotes: 18

Related Questions