Divide all the files present in the directory/folder to different thread for faster execution

Question

I got a program in which I have to write two thread that will perform same operation on the files present in a particular folder. That operation is to read all the files in that folder and then delete all of them present there. These two thread will have same operation but just to increase the process time I have to divide the files between two thread so that execution time is saved. I tried this way but it is not dividing the no. of files between threads but takes all the files and do the operation and if the file is deleted by one thread then other thread should not pick that file up. The code I wrote is:

public static void main(String[] args) throws InterruptedException {
    ExecutorService service = Executors.newFixedThreadPool(2);
    service.submit(new thread1());
    service.submit(new thread2());

    service.shutdown();
    service.awaitTermination(1, TimeUnit.DAYS);

    System.exit(0);
}

public static class thread1 implements Callable<Object> {

    @Override
    public Object call() throws Exception {
        t1();
        return null;
    }
}

public static class thread2 implements Callable<Object> {

    @Override
    public Object call() throws Exception {
        t1();
        return null;

    }
}

public static void t1() {

    Path myDir = Paths.get("D:/Astha/");
    File file = new File("D:/Astha/");
    boolean running = true;
    while (running) {
        try {
            WatchService watcher = myDir.getFileSystem().newWatchService();
            myDir.register(watcher, StandardWatchEventKinds.ENTRY_CREATE);

            WatchKey watckKey = watcher.take();

            List<WatchEvent<?>> events = watckKey.pollEvents();
            for (@SuppressWarnings("rawtypes")
            WatchEvent event : events) {
                if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
                    System.out.println("Created: " + event.context().toString() + "By "
                            + Thread.currentThread().getName());
                }
            }
            if (file.exists()) {
                File[] files = file.listFiles();
                for (File f : files) {
                    if (f.delete()) {
                        System.out.println("Deleting the file: " + f.getName() + "By "
                                + Thread.currentThread().getName());
                    }
                }
            } else {
                System.out.println("No files in the folder");
            }

        } catch (Exception e) {
            System.out.println("Error: " + e.toString());
        }
    }

}

In this above program, I also need to apply lock on one thread so that no other thread can perform operation on that. How do I implement lock here?

Answer 1

Based on your (updated) code, you have two basic task. You have a "watch" task and you have a "process" task

WatcherService

The WatcherService basically takes a Path and a ExecutorService, it monitors the given path and creates new FileTask tasks

public class WatcherService implements Callable<Object> {

    private Path path;
    private ExecutorService service;

    public WatcherService(Path path, ExecutorService service) {
        this.path = path;
        this.service = service;
    }

    @Override
    public Object call() throws Exception {
        do {
            try {
                WatchService watcher = path.getFileSystem().newWatchService();
                path.register(watcher, StandardWatchEventKinds.ENTRY_CREATE);

                WatchKey watckKey = watcher.take();

                List<WatchEvent<?>> events = watckKey.pollEvents();
                for (@SuppressWarnings("rawtypes") WatchEvent event : events) {
                    WatchEvent<Path> we = (WatchEvent<Path>)event;
                    service.submit(new FileTask(we.context()));
                }
            } catch (IOException | InterruptedException exp) {
                exp.printStackTrace();
            }
        } while (true && !Thread.currentThread().isInterrupted());
        return null;
    }

}

FileTask

The FileTask takes a Path and performs some operation upon it

public class FileTask implements Callable<Object> {

    private Path path;

    public FileTask(Path file) {
        this.path = file;
    }

    @Override
    public Object call() throws Exception {
        File file = path.toFile();
        if (file.exists()) {
            if (file.delete()) {
                //...
            }
        }
        return null;
    }

}

Hooking it up

Basically, you create a ExecutorService and submit a WatcherService and let it run...

Path path = Paths.get("D:/Astha/");
ExecutorService service = Executors.newFixedThreadPool(3);
service.submit(new WatcherService(path, service));

This creates a pooled service of three threads, one for the watcher and two for the FileTasks

You may find that this still does not offer you any benefit, as the disk I/O won't allow multiple operations to carried out in parallel and will block until the first operation completes before the next can be carried out

Answer 2

You could put all Files to edit in a HashSet or Map.

private static volatile Set<String> filenames;

If one of your threads is able to get the next File, use a synchronized Method to deliver it.

public synchronized String getNextMessage() {
if(filenames.size()<1) {
  return null;
} else {
  final String result = filenames.get(0);
  filenames.remove(0);
  return result;
}

Instead of a String, you could also use File, URI or Path, depending on your needs.