How To Increase Elasticsearch 5.6.7 Write Performance

Question

I am trying to benchmark my Hot-Warm Structured Elasticsearch cluster.I did a lot tests. I followed many things. I think, my index rate still is not acceptable.

In my cluster, I have 13 hot servers which have SSD, 3 of them has NVMe. Relatively good CPUs. And 3 warm servers. In my test, I get max 250k primary shard ingest, 500k total shards. Since I use 1 replicaton count.

For me It means, 38k document insert for per server, Since I use 1 replica count, it means 19k document per server. For such a basic key value model, the index rate is too slow for me. When I see that I cannot benchmark my real model.

Is there any advice for me?

I did following advices

• bootstrap.memory_lock: true
• thread_pool.bulk.queue_size: 100000 ( at the beginning it was 200, Then I tested 20000. now 100000)
• thread_pool.index.queue_size: 100000 ( at the beginning it was 200)
• Basic key value model. 1 replica, increased refresh interval, high or low shard count. Disabled _source and _all fields. My model is below. Auto generated ids.

• I ingest data using bulk processor. I tested changing document count and concurrency in the client side. I used multiple powerful servers to ingest data. My clients has hot server ips to ingest data, not warm servers.

  { "template": "test01*", "settings": { "index": { "routing": { "allocation": { "require": { "box_type": "hot" } } }, "refresh_interval": "600s", "number_of_shards": "26", "number_of_replicas": "1" } }, "mappings": { "test01": { "_source": { "enabled": false }, "_all": { "enabled": false }, "dynamic": false, "properties": { "rt": { "type": "keyword" }, "con": { "type": "float" } } } } }

Answer 1

a little advice, index setting should add index.routing.allocation.total_shards_per_node to control your index shards evenly distributed across nodes.