CODEWITHSUNDEEP
chroniclechronicle-queue
AmbGup
AmbGup

Reputation: 771

Chronicle Queue - Read from last read position and delete files if read by all consumers

I am using Chronicle 4.5.27 for writing and reading market data. I have got a single writer but multiple readers. Development OS is Windows followed by Linux for Prod deployment.

How to implement following uses cases?

  1. How to start reading queues form last read position ? e.g. if reader has read 15 records from a file which has 100 records and crashed/stopped how to start reading from 16th record from next restart? Is there an inbuilt durable support in CQ?
  2. Delete files which are read by all consumers to save disk space.

For this I have implemented but seems the files are not deleted on windows due to some open issue. Is there any built support in CQ where files can be deleted only if proceeded by all interested consumers? 

public static long readMarketData(String pathForMarketDataFile, long indexFrom) {
SingleChronicleQueue queue = SingleChronicleQueueBuilder.binary(pathForMarketDataFile).rollCycle(RollCycles.MINUTELY).storeFileListener(new StoreFileListener() {
    @Override
    public void onReleased(int i, File file) {
        System.out.println("File is not in use and is ready for deletion: " + file.getName());
        try {
            file.delete();
        } catch (IOException e) {
            e.printStackTrace();
        }
        ;
        System.out.println("File deleted:  " + file.getName() );
    }

    @Override
    public void onAcquired(int cycle, File file) {
        System.out.println("File is use for reading: " + file.getName());
    }
}).build();

I have read few blogs and posts on this topic e.g.

https://vanilla-java.github.io/2016/03/29/Microservices-in-the-Chronicle-world-Part-4.html

https://groups.google.com/forum/#!topic/java-chronicle/0Nz5P-nvLgM

But still want to know if any one has implemented this use case.

Upvotes: 2

Views: 1958

Answers (3)

Zheng
Zheng

Reputation: 11

According to the documentation, the Restartable Tailers from https://github.com/OpenHFT/Chronicle-Queue#restartable-tailers won't help if the reading was interrupted by error or crash.

When the chronicle queues are created, you can set the rollCycle. The file names will contain the date and hour (UTC time), then you can avoid reading the files that were created more than 1 hour ago if the rollCycle is set to HOURLY.

Upvotes: 0

Akash Gupta
Akash Gupta

Reputation: 21

1) If you look at the documentation https://github.com/OpenHFT/Chronicle-Queue#restartable-tailers , Naming your tailer will take care of this functionality automatically.

You need to give unique name to each of your reading tailer, such as 'a' given in the below line of code 

 ExcerptTailer atailer = cq.createTailer("a");

The tailer store their index within the Queue itself and this index is maintained. When your reading service gets restarted, the index will get picked & tailer continue from last read position.

2) When you call file.delete(), the file does not gets deleted on windows & functionality is working fine on Linux. Windows locks files that are currently in use by the program. You cannot delete the files which your application is currently using.

There are other solutions mentioned in other threads Java 'file.delete()' Is not Deleting Specified File

Upvotes: 1

Mark Price
Mark Price

Reputation: 326

Tracking of the consumer high-water mark is performed using MessageHistory, however, this requires that your consumers are also writing output to a chronicle queue (essentially storing the consumer read-sequence in the output queue).

Alternatively, you would need to implement your own mechanism for recording the highest sequence (index) each consumer has seen.

In terms of deleting files, there may be other processes holding open file handles to the queue files. If reader-A is no longer using queue file 15.cq4, then your code will attempt to call file.delete(), but reader-B may still have a reference to that file, stopping it from being deleted.

A more robust policy would be to have some sort of event from each reader to another service/process that is responsible for deleting the files once all readers have finished processing them.

Upvotes: 2

Related Questions