CODEWITHSUNDEEP
javalinux
Grumblesaurus
Grumblesaurus

Reputation: 3119

Best way to watch a large number of directories?

I have written a music player in Java/JavaFX that (among other things) watches the user's music library folder for any changes and updates the program's library data if there is a change on the filesystem.

I accomplish this with a SimpleFileVisitor and a WatchService. The SimpleFileVisitor recurively walks through the target folder heirarchy, and registers each folder with the WatchService. Full code below.

On my Ubuntu 18.04, if I try to register a number of folders that each have a lot of subfolders in this way, I eventually get the error java.io.IOException: User limit of inotify watches reached.

It turns out that the Linux kernel limits the number of inotify watches that can be done in the file /proc/sys/fs/inotify/max_user_watches. The common default is 8192. This limit is for every program run by the user, and there are other popular programs which use a lot, like DropBox.

My library potentially wants a lot as well. For the sort of user who might use this program, 1000 - 25000 folders that want to be watched is a reasonable estimate. I would expect most users to fall in the 2000 - 5000 range.

A solution is to notify the users of the error and tell them to increase the max_user_watches number in the kernel, which can be done, but I don't like putting that burden on the user if it can be avoided.

Is there another good way to be notified of changes in a folder hierarchy in java without overloading the inotify number on Linux?

My full watcher code can be found here, starting on line 545: https://github.com/JoshuaD84/HypnosMusicPlayer/blob/55da2819a3862d5c2a14797a9ce45e9171f7076e/src/net/joshuad/hypnos/Library.java#L545

You can also see the main walker code here:

private void watcherRegisterAll ( final Path start ) {
    try {
        Files.walkFileTree( 
            start, 
            EnumSet.of( FileVisitOption.FOLLOW_LINKS ), 
            Integer.MAX_VALUE,
            new SimpleFileVisitor <Path>() {
                @Override
                public FileVisitResult preVisitDirectory ( Path dir, BasicFileAttributes attrs ) throws IOException {
                    WatchKey key = dir.register( 
                        watcher, 
                        StandardWatchEventKinds.ENTRY_CREATE, 
                        StandardWatchEventKinds.ENTRY_DELETE, 
                        StandardWatchEventKinds.ENTRY_MODIFY 
                    );

                    keys.put( key, dir );
                    return FileVisitResult.CONTINUE;
                }
            }
        );

    } catch ( IOException e ) {
        LOGGER.log( Level.INFO, "Unable to watch directory for changes: " + start.toString() );
    }
}

Upvotes: 1

Views: 763

Answers (1)

Uku Loskit
Uku Loskit

Reputation: 42040

Your approach is the correct one, and inotify is the most performant way to do this. If you run into this issue, you could just prompt the user to increase limit on their behalf to provide a better user experience (you have to perform a privilege escalation by asking for password for this).

Upvotes: 1

Related Questions