CODEWITHSUNDEEP
couchbasenosql
aneez
aneez

Reputation: 693

Multiple inserts in Couchbase failing

Couchbase newbie question:

I am trying to insert 1 million records in couchbase, but I see that nearly about 0.5 million records get inserted (Admin console shows 517365 as Item Count). Also, from the admin GUI, I can only see 1000 records (10 pages of 100 records each)

Am wondering where rest of the records are vanishing !

1)Can somebody help me with this ?

2)Which log file I should be looking at to find insertion failure errors ?

I suspect Couchbase has a internal queue. Once it gets full, further requests are dropped. If yes, then how to configure the queue size ?

PS: I tried looking into the logs C:\Program Files\Couchbase\Server\var\lib\couchbase\logs, but couldn't figure out anything.

public class Test {
    public static void main(String[] args) {
        ArrayList<URI> nodes = new ArrayList<URI>();
        String cbUrl = "http://127.0.0.1:8091/pools";
        String dbName = "deafult";
        CouchbaseClient client = null;
        try {
            nodes.add(URI.create(cbUrl));

            client = new CouchbaseClient(nodes, dbName, "");

            insertRecords(client);

            System.out.println("Test Over");

        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            // client.shutdown();
        }
    }

    public static void insertRecords(CouchbaseClient client) throws Exception {
        int num = 1000000;

        for (int n = 1; n <= num; n++) {
            System.out.println("Adding: " + n);
            client.set(n + "", 0, n + "");
        }
    }
}

Upvotes: 1

Views: 488

Answers (2)

mikewied
mikewied

Reputation: 5323

The set operation in the Couchbase Java SDK is asynchronous. This means once the call returns there is no guarantee that you've even sent the operation to Couchbase since it may not have even been written to the network buffer yet. In order to make sure the operation has completed you need to call the get() function on the object (which is a Future) returned by the set() API.

In other words replace this line:

client.set(n + "", 0, n + "");

with this one:

client.set(n + "", 0, n + "").get();

Upvotes: 2

DaveR
DaveR

Reputation: 9640

To expand on @mikewied's answer, to check that all 1,000,000 set operations have completed without having to call .get() explicitly on each one (and hence converting the calls from async to sync), you need to add a listener to each set which tracks how many of your operations have completed.

There's a nice example of how to do this in the blog post announcing Couchbase Java SDK 1.2 :-

final CountDownLatch latch = new CountDownLatch(100);
for (int i = 0; i < 100; i++) {
  OperationFuture<Boolean> future = client.set("key-" + i, "value");
  future.addListener(new OperationCompletionListener() {
    @Override
    public void onComplete(OperationFuture<?> future) throws Exception {
      latch.countDown();
    }
  });
}
latch.await();

You create a CountDownLatch, initialised to how many documents you are set()ing, then register a listener which is called on completion of each set (but note the sets are still asynchronous). At the end you then call await() on the latch to ensure that all set operations have completed before continuing.

This approach is described in more detail in the Understanding and Using Asynchronous Operations section of the Couchbase Java SDK Developer guide, along with a more compact syntax if you're using Java 8.

Upvotes: 0

Related Questions