CODEWITHSUNDEEP
javamultithreadingfile-processing
leo
leo

Reputation: 35

how to process plenty of files by multi-thread in java

Now I have plenty of JSON files on local, I need read these files and transfer them to JSONObject then do some business logical, such as find out the child-node then return, count some special value then return.

As the single file, I have finished the code of read the JSON file-->transfer it to JSONObject-->do the search of child-node/ count some special value.

But if when there was plenty of JSON files, how can I do this job? I believe multi-thread could resolve my problem, but how can I collect every thread's result and finally get one result?

By the way, if I need create a thread pool and how ?

Upvotes: 0

Views: 140

Answers (1)

Peter Lawrey
Peter Lawrey

Reputation: 533530

You can try multi-threading it using Streams. There is no guarantee this will be faster, or if your tasks use lots of memory, you could run out of memory but this is a simple way to try this.

List<Result> results= Files.list(path).parallel()
                           .map(p -> turnFileIntoJSONObject(p))
                           .map(json -> process(json))
                           .collect(Collectors.toList());

This will distribute the files to be processed across your available CPUs and collect the results at the end.

You can also drop the .parallel() if it is not faster for you.

Upvotes: 1

Related Questions