How to verify file is finished being written before attempting to process it?

Question

Hard to give a good title to my problem but it is as follows. First I'm doing this on windows and it could possibly be used on a linux box also so I'd need the fix to work on both systems. I am monitoring a directory for new files. I basically looking at the directory's files and comparing them over and over and only processing the new files. Problem is I keep getting an error where the file isn't finished being written before I attempt to process.

public class LiveDetectionsProvider extends DetectionsProvider {
protected LiveDetectionsProvider.MonitorDirectory monitorDirectory = null;
protected TimeModel timeModel = null;
private ArrayList<String> loaded = new ArrayList();
private File topLayerFolder = null;
public LiveDetectionsProvider(String directory, String id) {
    super(directory, id);
    timeModel = super.timeModel;
}
/**
 * Initialize the data provider.
 */
public void initialize() {
    try {
        topLayerFolder = new File(directory);
        File[] dir = topLayerFolder.listFiles();
        for (File file : dir) {
            loaded.add(file.getName());
        }
        monitorDirectory = new MonitorDirectory();
        monitorDirectory.execute();
    }
    catch (Exception ex) {
        Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Failed to read detection\n{0}", ex.getMessage());
    }
    super.initialize();
}
/**
 * Un-initialize the data provider.
 */
public void uninitialize() {
    super.uninitialize();
    if (monitorDirectory != null) {
        monitorDirectory.continuing = false;
    }
}
/**
 * The class that is used to load the detection points in a background
 * thread.
 */
protected class MonitorDirectory extends SwingWorker<Void, Void> {
    public boolean continuing = true;
    /**
     * The executor service thread pool.
     */
    private ExecutorService executor = null;
    /**
     * The completion service that reports the completed threads.
     */
    private CompletionService<Object> completionService = null;
    @Override
    protected Void doInBackground() throws Exception {
        int count = 0;
        executor = Executors.newFixedThreadPool(1);
        completionService = new ExecutorCompletionService<>(executor);
        while (continuing && topLayerFolder != null) {
            File[] dir = topLayerFolder.listFiles();
            Thread.sleep(10);
            ArrayList<File> filesToLoad = new ArrayList();
            for (File file : dir) {
                if (!loaded.contains(file.getName())) {
                    long filesize = 0;
                    boolean cont = true;
                    while (cont) {
                        if (file.length() == filesize) {
                            cont = false;
                            Thread.sleep(3);
                            filesToLoad.add(file);
                        }
                        else {
                            filesize = file.length();
                            Thread.sleep(3);
                        }
                    }
                    Thread.sleep(3);
                }
            }
            for (File file : filesToLoad) {
                timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), true);
                completionService.submit(Executors.callable(new ReadDetection(file, false)));
                while (completionService.take() == null) {
                    Thread.sleep(2);
                }
                loaded.add(file.getName());
                count++;
                Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Detection Message Count:" + count);
            }
            detectionsModel.fireStateChanged(DetectionsModel.CHANGE_EVENT_DETECTIONS);
            timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), false);
        }
        return null;
    }
}
}

The file is processed at the line with

completionService.submit(Executors.callable(new ReadDetection(file, false)));

The file at this point still hasnt finished being written and thus fails. I've tried sleeping my thread to slow it down, and I've tried verifying the file size hasn't changed. My test case for this is I'm unzipping a tar file which contains tons of 1,000 KB files.

Answer 1

First yes it compiles look a the solution I posted in relation to the code In my question. You substitute

For(File file: dir){
while(!file.renameTo(file)){
Thread.sleep(1)
}
// In my code I check to see if the file name is already in the list which 
// contains files that have been previously loaded if its not I add it to a list
// of files to be processed
}

in for

        for (File file : dir) {
            if (!loaded.contains(file.getName())) {
                long filesize = 0;
                boolean cont = true;
                while (cont) {
                    if (file.length() == filesize) {
                        cont = false;
                        Thread.sleep(3);
                        filesToLoad.add(file);
                    }
                    else {
                        filesize = file.length();
                        Thread.sleep(3);
                    }
                }
                Thread.sleep(3);
            }
        }

sorry I forgot to put comment tags // in on the line that said do what every you need to do here.

What it does is it looks at each file in the directory and checks to see if you can rename it if the rename fails it sleeps and continues checking till it sucessfully is able to rename at which point you can do what you need to with the file which in my case was everything after the for loop that was replaced. I'm curious why my awnser was viewed as sub par deleted and locked. This solution does work and solved my problem and would anyone else who's having the same issue attempting to process a file that still being written or copied to a directory thats being monitored for changes.

Answer 2

Use a "flag file": once file.txt is "finished", indicate this by creating a file.flg - consuming process should wait for .flg to appear.

Answer 3

Usually I solve this issue by create a temporary file while the file is being written. Once finish I rename the file and only the renamed file can be process.