CODEWITHSUNDEEP
elasticsearch
Rafael
Rafael

Reputation: 1223

Increase queue capacity in ElasticSearch

Elastic version 7.8

I'm getting an error when running this code for thousands of records:

var bulkIndexResponse = await _client.BulkAsync(i => i
                .Index(indexName)
                .IndexMany(bases));

                if (!bulkIndexResponse.IsValid)
                {
                    throw bulkIndexResponse.OriginalException;
                }

It eventually crashes with the following error:

Invalid NEST response built from a successful (200) low level call on POST: /indexname/_bulk
# Invalid Bulk items:
operation[1159]: index returned 429 _index: indexname _type: _doc _id:  _version: 0 error: Type: 
es_rejected_execution_exception Reason: "Could not perform enrichment, enrich coordination queue at 
capacity [1024/1024]"

I would like to know how this enrich coordination queue capacity can be increased to accommodate continuous calls of BulkAsync with around a thousand records on each call.

Upvotes: 2

Views: 3375

Answers (2)

hamid bayat
hamid bayat

Reputation: 2179

you can check what thread_pool is getting full by /_cat/thread_pool?v and increase the queue (as ninja said) in elasticsearch.yml for each node.

but increasing queue size affect heap consumption and subsequently maybe it would affect performance.

when you get this error it may have two reason. first you are sending large bulk request. try to decrease the bulk request under 500 or lower. second you have some performance issue. try to find and solve the issue. maybe you should add more node to your cluster.

Upvotes: 2

Amit
Amit

Reputation: 32386

Not sure what version you are, but this enrich coordination queue seems to be the bulk queue and you can increase the queue size(these are node specific) by changing the elasticsearch.yml of that node.

Refer threadpools in ES for more info.

Upvotes: 1

Related Questions