CODEWITHSUNDEEP
javarx-javacouchbase
M. Henrikh
M. Henrikh

Reputation: 71

Bulk read Couchbase documents

I want to asynchronously read a number of documents from a Couchbase bucket. This is my code:

JsonDocument student = bucketStudent.get(studentID);

The problem is for a large data file with a lot of studentIDs, it would take a long time to get all documents for these studentIDs because the get() method is called for each studentID. Is it possible to have a list of studentIDs as input and return an output of a list of students instead of getting a single document for each studentID?

Upvotes: 0

Views: 2122

Answers (4)

best wishes
best wishes

Reputation: 6624

It is possible now. The java sdk gives capability to do multi get. It is present in 2 flavours

  1. Async bulk get

  2. N1Q1 query. Does not work with binary documents

The Couchbase document suggests to use Async bulk get, But that has additional dependency on reactive java client. you can see official documentation here.

There are several tutorial explaining the usage. link .

Here is how a sample get would look like, with java sdk 3.2

    List<String> docsToFetch = Arrays.asList("airline_112", "airline_1191", "airline_1203");

    Map<String, GetResult> successfulResults = new ConcurrentHashMap<>();
    Map<String, Throwable> erroredResults = new ConcurrentHashMap<>();

    Flux.fromIterable(docsToFetch).flatMap(key -> reactiveCollection.get(key).onErrorResume(e -> {
        erroredResults.put(key, e);
        return Mono.empty();
    }).doOnNext(getResult -> successfulResults.put(key, getResult))).last().block();

source .

Upvotes: 0

Madhu Bhat
Madhu Bhat

Reputation: 15173

Retrieving multiple documents using the document IDs is not supported by default in the Couchbase Java SDK. To achieve that you'll need to use a N1QL query as below

SELECT S.* FROM Student S USE KEYS ["StudentID1", "StudentID2", "StudentID3"]

which would return an array of Documents with the given IDs. Construct the query with com.couchbase.client.java.query.N1qlQuery, and use either of below to execute

If you're using Spring's CouchbaseTemplate, you can use the below

List<T> findByN1QL(com.couchbase.client.java.query.N1qlQuery n1ql,
                          Class<T> entityClass)

If you're using Couchbase's Java SDK, you can use the below

N1qlQueryResult query(N1qlQuery query)

Note that you'll need an index on your bucket to run N1QL queries.

Upvotes: 1

6160
6160

Reputation: 1002

AFAIK couchbase SDK does not have a native function for a bulk get operation.

The node.js SDK has a getMulti method, but it's basically an iteration over an array and then get() is fired for each element.

I've found in my applications that the key-value approach is still faster than the SELECT * on a primary index but the N1QL query is remarkably close (on couchbase 5.x).

Just a quick tip: if you have A LOT of ids to fetch and you decide to go with the N1QL queries, try to split that list in smaller chunks. It's speeds up the queries and you can actually manage your errors better and avoid getting some nasty timeouts.

Upvotes: 1

Johan Larson
Johan Larson

Reputation: 1890

If you are running a query node, you can use N1QL for this. Your query would look like this:

SELECT * FROM myBucket USE KEYS ["key1", "key2", "key3"]

In practice you would probably pass in the array of strings as a parameter, like this:

SELECT * FROM myBucket USE KEYS ?

You will need a primary index for your bucket, or queries like this won't work.

Upvotes: 1

Related Questions