CODEWITHSUNDEEP
couchbasecouchbase-viewcouchbase-java-api
Onkaar Singh
Onkaar Singh

Reputation: 10463

Large number of KV operations(insert, upsert) on a couchbase bucket leads to concurrent.timeout.Exception

I am trying to insert/upsert documents into couchbase database using Java JDK client 2.4.7 and couchbase server 5.0.1. The couchbase cluster is setup on AWS EC2 with 6 data nodes of type m4.2xLarge and two index/query nodes of type m4.xLarge.

The documents count(records) to be inserted into couchbase is about 6 millions. These records are group together into chunks of 10K each. 20 worker threads are performing insert/upsert operation for each chunk of 10K by iterating over each record using a for loop. The time taken for insertion of each chunk keeps on increasing over a regular interval, eventually leading to the following exception:

java.lang.RuntimeException: java.util.concurrent.TimeoutException
    at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:77)
    at com.couchbase.client.java.subdoc.MutateInBuilder.execute(MutateInBuilder.java:280)
    at com.couchbase.client.java.subdoc.MutateInBuilder.execute(MutateInBuilder.java:103)
    at com.onkaar.XXXXXXXX.kernel.infrastructure.data.CouchbaseSaveAccountRepository.saveBatch(CouchbaseSaveAccountRepository.java:56)
    at com.onkaar.XXXXXXXX.kernel.storage.StorageTask.executeTask(StorageTask.java:38)
    at com.onkaar.XXXXXXXX.kernel.storage.StorageTask.run(StorageTask.java:63)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.TimeoutException

Following are the environment configuration I am using for connecting to the couchbase cluster: Couchbase IO Pool Size = 5; Computation pool Size = 5; KV Endpoint Count = 3; KV Timeout = 5000; Connection Timeout = 10000; Worker Threads = 20;

Can you please suggest optimal configuration for this use case.

Upvotes: 1

Views: 662

Answers (0)

Related Questions