CODEWITHSUNDEEP
javaperformancejava-8java-streamfilestream
miserable
miserable

Reputation: 727

Disk reading operation is performing very slow | Java Stream

I need to read images from a folder and generate checksum for them. There are about 330760 images. Following is the code: 

package com.test;

import java.io.FileInputStream;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;

import org.apache.commons.codec.digest.DigestUtils;

public class FileTest2 {

    private void readFiles() throws IOException {
        try (Stream<Path> filePathStream = Files
                .walk(Paths.get("d:\\codebase\\images"))) {
            filePathStream.parallel().forEach(filePath -> {
                String checksumSHA256 = "";
                try {
                    checksumSHA256 = DigestUtils.sha384Hex(new FileInputStream(filePath.toString()));
                } catch (IOException e) {
                    e.printStackTrace();
                }
                if (Files.isRegularFile(filePath)) {
                    System.out.println(checksumSHA256);
                    System.out.println(filePath);
                    System.out.println("\n");

                }
            });
        }
    }

    public static void main(String[] args) throws IOException {
        long startTime = System.currentTimeMillis();
        FileTest2 fileTest = new FileTest2();
        fileTest.readFiles();
        long endTime = System.currentTimeMillis();
        System.out.println("Total Time took: " + (endTime - startTime) / 1000);
    }
}

It took about 36 minutes.

System configuration:

Cores: 8
Memory: 32 GB (15-17 GB is free). Rest of the memory is being used by another server.

36 minutes are too much. Is there a way to improve performance?

Upvotes: 0

Views: 447

Answers (1)

David Soroko
David Soroko

Reputation: 9086

As others pointed out you do not terminate the executor. To see the actual times run the following

public static void main(String[] args) throws Exception {
    long startTime = System.currentTimeMillis();

    FileTest fileTest = new FileTest();
    fileTest.readFiles();

    long endTime = System.currentTimeMillis();
    System.out.println("Total Time took: "+ (endTime-startTime)/1000);
}

Note: at least from the bit of code you posted there is no reason to use an ExecutorService

Upvotes: 2

Related Questions