CODEWITHSUNDEEP
multithreadingclojureparallel-processingfuture
APatrikMs94
APatrikMs94

Reputation: 299

Clojure computing array in parallel

I need some help. I have on array(sequence) in size of 4000 elements. I need to compute each part(1000 elements is one part) by four different functions. On function computing one part at least 10 sec. Another functions computing their parts in 1 sec. I need to do it at list 10 times. So, of course I want to do it in parallel. I'm trying to do it using clojure future, but it steel take computing time like a sequence program. At first I think, that's because when i'm using one array and send data to each function each future thread can get access sequental. How can I do this computing in parallel?

here s a simple example of code:

(defn funct [t n]
 (let [ a (future (fun1 (fun_take_i_part_array n 1)))
        b (future (fun2 (fun_take_i_part_array n 2)))
        c (future (fun3 (fun_take_i_part_array n 3)))
        d (future (fun4 (fun_take_i_part_array n 4)))
      ]
 (concatenate @a @b @c @d)
 )

)

So I this future-kind program take longest than sequence for 1 sec.

Upvotes: 2

Views: 235

Answers (1)

Terminus
Terminus

Reputation: 949

The array access is never sequential unless these are not arrays but lazy sequences. It does not need to be because arrays are immutable - that is the distinguishing feature of Clojure. Your slowdown does not come from that.

Your idea of parallelisation is correct, look at this:

(defn fibonacci [n] (case n 1 1 2 1 (+ (fibonacci (- n 2)) (fibonacci (- n 1)))))
(defn parallel-fib [] 
        (let [s1 (future (fibonacci 34)) 
              s2 (future (fibonacci 35)) 
              s3 (future (fibonacci 36)) 
              s4 (future (fibonacci 33))] 
 [@s1 @s2 @s3 @s4]))

 user=> (time (fibonacci 35))
 "Elapsed time: 1514.663902 msecs"
 9227465

 user=> (time (fibonacci 36))
 "Elapsed time: 2403.761528 msecs"
 14930352

 user=> (time (parallel-fib))
 "Elapsed time: 2552.572043 msecs"
 [5702887 9227465 14930352 3524578]

It is clear that the parallel executed futures, running on a 4-core machine, yield time close to the most expensive computation and not to the sum of computation times.

Therefore, I see no obvious reasons why your code experiences times closer to sequential execution. The reasons that come to my mind might be:

  1. fun1 (or any other of those funs) could be a parallel function by itself (e.g. using pmap) consuming all of your cores. Then, the CPU would get crammed and no speedup would be observed.

  2. Somehow 'concatenate' is eating a lot of CPU to put the results together. This should not be a major hassle as if you have used standard Clojure 'concat', it produces a lazy sequence which should incur its access cost only after it's being accessed. Not to mention the access cost should be very low.

  3. If the computations inside futures are very simple, no good scaling could be archived either. We know it is not true, though, because you have already mentioned it takes fun1 10 seconds to complete.

It could help if you could send me the whole code. My email: contact [at] spinney.io.

Upvotes: 1

Related Questions