CODEWITHSUNDEEP
javahazelcastcircular-buffer
Andy Dufresne
Andy Dufresne

Reputation: 6190

How to speed up data imports from ring buffers in a java client?

We have a third party application producing data at a very high speed and storing it in multiple ring buffers (hazelcast cluster).

The consumer application (another third party app) uses this java hazelcast library to read ring buffer data. It calls final byte[] item = ringbuffer.readOne(sequence) api for reads. This turns out to be very slow. The consumer takes more than a day to read all data during peak loads.

What are general strategies to concurrently read data from hazelcast cluster? Are there any java references that I can look into?

Thanks.

Upvotes: 0

Views: 75

Answers (1)

Nikhil Pandit
Nikhil Pandit

Reputation: 323

I have never used Hazelcast but as a Java developer got interest in this question. I checked the Hazelcast documentation page and found a section where they refer to reading data in batches, you can try that.

Below method will help you in acheiving that:

CompletionStage<ReadResultSet<E>> readManyAsync(
    long startSequence,
    int minCount,
    int maxCount,
    IFunction<E, Boolean> filter);

Meaning of the arguments of this method are explained in more detail on the reference link of documentation.

long sequence = rb.headSequence();
for(;;) {
    CompletionStage<ReadResultSet<String>> f = rb.readManyAsync(sequence, 1, 10, null);
    CompletionStage<Integer> readCountStage = f.thenApplyAsync(rs -> {
        for (String s : rs) {
            System.out.println(s);
        }
        return rs.readCount();
    });
    sequence += readCountStage.toCompletableFuture().join();
}

This one also provides a filter which can be used to filter data while fetching from ringbuffer. Where as in readOne method you fetch the data and filter after it.

Reference link: https://docs.hazelcast.com/hazelcast/5.3/data-structures/ringbuffer#reading-batched-items

Upvotes: 1

Related Questions