CODEWITHSUNDEEP
elasticsearchthreadpool
Yallkun M
Yallkun M

Reputation: 33

show and change parameter threadpool.bulk.queue_size

I am using an elasticsearch version 6.7.2 and I have error 429

{
    "error": {
        "root_cause": [
            {
                "type": "remote_transport_exception",
                "reason": "[localhost][localhost:9300][indices:data/write/bulk[s]]"
            }
        ],
        "type": "es_rejected_execution_exception",
        "reason": "rejected execution of processing of [682604930][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[hboauth2-2020.04.20-000113][0]] containing [index {[hboauth2][_doc][yRuEnHEBn6z33bhuCGjZ], source[{\"Action\":\"Service UpdateUserStatus Start\",\"ClientID\":\"\",\"Code\":0,\"Component\":\"UpdateUserStatus\",\"CustomMessage\":\"\",\"Data\":\"null\",\"LogDate\":1587468830939,\"Message\":\"{OK false}\",\"Type\":\"Info\",\"UserID\":\"4c220ab2-282d-466f-a9f5-da0e59b7b803\"}]}], target allocation id: wZFZUPi5TnSYFg-BNonHug, primary term: 1 on EsThreadPoolExecutor[name = localhost/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@744831b0[Running, pool size = 16, active threads = 16, queued tasks = 224, completed tasks = 264136589]]"
    },
    "status": 429
}

I found a solution to the problem, but I can not find the parameter threadpool.bulk.queue_size in my version. could you please help me how to find and change this parameter

Upvotes: 2

Views: 10298

Answers (2)

Amit
Amit

Reputation: 32386

As shown in the official example, you can change the queue size of bulk thread pool using the below config in your elasticsearch.config or update the cluster setting.

Note, write thread pool is used for bulk request as documented in the same link:

write For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.

thread_pool:
    write:
        size: 30. --> no if threads in your case, its 16 as shown in exception
        queue_size: 400 --> here you can add `queue_size`

Instruction on changing these settings also mentioned

Changing a specific thread pool can be done by setting its type-specific parameters; for example, changing the number of threads in the write thread pool:

Edit: As pointed out by @val in the comment section, Adding disclaimer about changing these default settings.

Please understand the technicalities of these settings and the impact, as these are often not advisable to change, more info in this official blog

Upvotes: 2

Yallkun M
Yallkun M

Reputation: 33

i was searching how i can change thread_pool.write.queue_size and find 2 solutions first 

curl -XPUT  _cluster/settings -d '{
    "persistent" : {
        "thread_pool.write.queue_size" : <new_size>
    }
}'

second write to elasticsearch.yaml paramater thread_pool.write.queue_size

all solutions are valid for my version 6.2.7?

Upvotes: 1

Related Questions