CODEWITHSUNDEEP
javamultithreadinginputstreamexecutorservice
user19772950
user19772950

Reputation:

Executor service with Inputstream, read txt file in multiple threads

I have searched for this everywhere but could not find a suitable solution for my case.

I have a program that takes txt file as program argument, create output txt file, check type of the file

-i /Users/me/test.txt -o /Users/me/output.txt -t TEST

I have a processor to process/extract info from the txt file, check if the extracted data exists via api call to the service

I have relatively large txt file to read and check line by line. I would like to read it with executor service.

I have a processor to process the input file, extract data:

public class FileParser {
  private String line;

  public List<FileInfo> processFile(InputStream inputStream) throws IOException {
    List<FileInfo> resultSet = new ArrayList<>();
    try (Scanner sc = new Scanner(inputStream, "UTF_8")) {
        line = sc.nextLine();
        while (line != null) {
            if (!line.trim().startWith("*") && !line.isEmpty()) {
                String number = line.subString(1, 4);
                String id = line.subString(5, 10);

                FileInfo fileInfo = new FileInfo(number, id);
                resultSet.add(fileInfo);
            }
            if (sc.hasNextLine()) {
                line = sc.nextLine();
            } else {
                break;
            }
        }
        if (sc.ioException() != null) {
            throw sc.ioException();
        }
    }
    return resultSet;
  }
}

In main, I have:

public class ValidatorApplication {
    public static void main(String[] args) {
        ValidatorApplication validator = new ValidatorApplication();
        // parse program argument like above -i, -o, -t
        validator.parseArguments(arg);

        // output txt file model
        Report report = new Report();
        FileParser fileParser = new FileParser();
        // service for API call
        ApiService apiService = new ApiService();
        // Output stream - how the output txt file look like
        ReportGenerator reportGenerator = new ReportGenerator();
        report.setFileName(validator.outputFilePath);

        try (InputStream inputStream = new FileInputStream(validator.inputFilePath)) {
            if (validator.type.equalsIgnoreCase("TEST")) {
                List<FileInfo> fileInfoList = fileParser.processFile(inputStream);
                for(FileInfo fileInfo : fileInfoList) {
                    if (!apiService.isDataExist(fileInfo)) {
                        report.getUnmatched().add(fileInfo);
                    }
                }
            }

            reportGenerator.generator(report);

        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

This program works fine. I am having trouble figuring out how to apply ExecutorService on top of what is here. I just want some basic thread pools to process the file.

Upvotes: 0

Views: 593

Answers (1)

queeg
queeg

Reputation: 9463

  • Reading from an input stream is a sequential task. Trying to do that via several threads is not feasible.
  • On the other hand, running the check via the API service might take comparatively long and low resources on your end. So you'd like to parallelize that.
  • Finally writing the result to the output stream is sequential again.

So what you might want to do is:

  • have one thread that reads the input file and sends 'process line' jobs into the executor service
  • have an executor service with whatever number of threads to process the queued jobs in parallel. Results will be written into some result buffer
  • have one thread to check the result buffer and write any result into the output stream

With that solution you can parallel process the records. But the resulting output file may have a different sequence of lines.

Upvotes: 2

Related Questions