CODEWITHSUNDEEP
clojureparallel-processing
nha
nha

Reputation: 18005

Clojure structure multiple calculation/writes to work in parallel

Let's say I have the following code :

(defn multiple-writes []
  (doseq [[x y] (map list [1 2] [3 4])] ;; let's imagine those are paths to files
     (when-not (exists? x y) ;; could be left off, I feel it is faster to check before overwriting
       (write-to-disk! (do-something x y)))))

That I call like this (parameters omitted) :

   (go (multiple-writes))

I use go to execute some code "in the background", but I do not know if I am using the right tool here. Some more information about those functions :

So I would like to speed up things by executing (write-to-disk! (do-something x y)) in parallel to go as fast as possible. But I don't want to overload the system at all, since this is not a high-priority task.

How should I go about this ?

Note : despite the title, this is not a duplicate of this question since I don't want to restrict to 3 threads (not saying that the answer can't be the same, but I feel this question differs).

Upvotes: 2

Views: 200

Answers (2)

NielsK
NielsK

Reputation: 6956

Take a look at the claypoole library, which gives some good and simple abstractions filling the void between pmap and fork/join reducers, which otherwise would need to be coded by hand with futures and promises.

With pmap all results of a parallel batch need to have returned before the next batch is executed, because return order is preserved. This can be a problem with widely varying processing times (be they calculation, http requests, or work items of different "size"). This is what usually slows down pmap to single threaded map + unneeded overhead performance.

With claypoole's unordered pmap and unordered for (upmap and upfor), slower function calls in one thread (core) can be overtaken by faster ones on another thread because ordering doesn't need to be preserved, as long as not all cores are clogged by slow calls.

This might not help much in case of IO to one disk being the only bottleneck, but since claypoole has configurable thread pool sizes and functions to detect the number of available cores, it will help with restricting the amount of cores.

And where fork/join reducers would optimize CPU usage by work stealing, it might greatly increase memory use, since there is no option to restrict the amount of parallel processes without altering the reducer library.

Upvotes: 1

muhuk
muhuk

Reputation: 16085

Consider basing your design on streams or fork/join.

I would a single component that does IO. Every processing node can then send their results to be saved there. This is easy to model with streams. With fork/join, it can be achieved by not returning the result up in the hierarchy but sending it to eg. an agent.

If memory consumption is an issue, perhaps you can divide work even more. Like 100x100 patches.

Upvotes: 1

Related Questions