CODEWITHSUNDEEP
javafileout-of-memory
James
James

Reputation: 1001

OutOfMemory when list files in a directory

When I list files of a directory that has 300,000 files with Java, out of memory occurs. 

String[] fileNames = file.list();

What I want is a way that can list all files of a directory incrementally no matter how many files in that specific directory and won't have "out of memory" problem with the default 64M heap limit.

I have Google a while, and cannot find such a way in pure Java.
Please help me!!

Note, JNI is a possible solution, but I hate JNI.

Upvotes: 12

Views: 7347

Answers (5)

Mirko
Mirko

Reputation: 1552

The only possible solution for you is Java7 and then you sould use an iterator.

final Path p = FileSystems.getDefault().getPath("Yourpath");
Files.walk(p).forEach(filePath -> {
        if (Files.isRegularFile(filePath)) {
            //Do something with filePath
        }
});

Upvotes: 5

infect
infect

Reputation: 41

If you can write your code in Java 7 or up, then following is a good option.

Files.newDirectoryStream(Path dir)

Here is the java doc for the API.

Hope this helps.

Upvotes: 0

Ondra Žižka
Ondra Žižka

Reputation: 46876

Having 300 000 files in a dir is not a good idea - AFAIK filesystems are not good at having that many sub-nodes in a single node. Interesting question, though.

EDIT: THE FOLLOWING DOES NOT HELP, see comments.

I think you could use a FileFilter, reject all files, and process them in the filter.

        new File("c:/").listFiles( new FileFilter() {
            @Override   public boolean accept(File pathname) {
                processFile();
                return false;
            }
        });

Upvotes: 1

danben
danben

Reputation: 83270

I know you said "with the default 64M heap limit", but let's look at the facts - you want to hold a (potentially) large number of items in memory, using the mechanisms made available to you by Java. So, unless there is some dire reason that you can't, I would say increasing the heap is the way to go.

Here is a link to the same discussion at JavaRanch: http://www.coderanch.com/t/381939/Java-General/java/iterate-over-files-directory

Edit, in response to comment: the reason I said he wants to hold a large number of items in memory is because this is the only mechanism Java provides for listing a directory without using the native interface or platform-specific mechanisms (and the OP said he wanted "pure Java").

Upvotes: 5

Gennadiy
Gennadiy

Reputation: 18227

You are a bit out of luck here. In the least there will need to be created 300k strings. With an average length of 8-10 char and 2 bytes per char thats 6Mb in the minimum. Add object pointer overhead per string (8 bytes) and you run into your memory limit.

If you absolutely must have that many files in a single dir, which i would not recommend as your file system will have problems, your best bet is to run a native process (not JNI) via Runtime.exec. Keep in mind that you will tie yourself down to the OS (ls vs dir). You will be able to get a list of files as one large string and will be responsible for post processing it into what you want.

Hope this helps.

Upvotes: 2

Related Questions