Reputation: 343
I have the following Julia code and I would like to parallelize it.
using DistributedArrays
function f(x)
return x^2;
end
y = DArray[]
@parallel for i in 1:100
y[i] = f(i)
end
println(y)
The output is DistributedArrays.DArray[]
. I would like to have the value of y as follows: y=[1,4,9,16,...,10000]
Upvotes: 15
Views: 6914
Reputation: 7893
You can use n-dimensional distributed array comprehensions:
First you need to add some more processes, either local or remote:
julia> addprocs(CPU_CORES - 1);
Then you must use DistributedArrays
at every one of the spawned processes:
julia> @everywhere using DistributedArrays
Finally you can use the @DArray
macro, like this:
julia> x = @DArray [@show x^2 for x = 1:10];
From worker 2: x ^ 2 = 1
From worker 2: x ^ 2 = 4
From worker 4: x ^ 2 = 64
From worker 2: x ^ 2 = 9
From worker 4: x ^ 2 = 81
From worker 4: x ^ 2 = 100
From worker 3: x ^ 2 = 16
From worker 3: x ^ 2 = 25
From worker 3: x ^ 2 = 36
From worker 3: x ^ 2 = 49
You can see it does what you expect:
julia> x
10-element DistributedArrays.DArray{Int64,1,Array{Int64,1}}:
1
4
9
16
25
36
49
64
81
100
Remember it works with an arbitrary number of dimensions:
julia> y = @DArray [@show i + j for i = 1:3, j = 4:6];
From worker 4: i + j = 7
From worker 4: i + j = 8
From worker 4: i + j = 9
From worker 2: i + j = 5
From worker 2: i + j = 6
From worker 2: i + j = 7
From worker 3: i + j = 6
From worker 3: i + j = 7
From worker 3: i + j = 8
julia> y
3x3 DistributedArrays.DArray{Int64,2,Array{Int64,2}}:
5 6 7
6 7 8
7 8 9
julia>
This is the most julian way to do what you intended IMHO.
We can look at macroexpand
output in order to see what's going on:
Note: this output has been slightly edited for readability, T
stands for:
DistributedArrays.Tuple{DistributedArrays.Vararg{DistributedArrays.UnitRange{DistributedArrays.Int}}}
julia> macroexpand(:(@DArray [i^2 for i = 1:10]))
:(
DistributedArrays.DArray(
(
#231#I::T -> begin
[i ^ 2 for i = (1:10)[#231#I[1]]]
end
),
DistributedArrays.tuple(DistributedArrays.length(1:10))
)
)
Which basically is the same as manually typing:
julia> n = 10; dims = (n,);
julia> DArray(x -> [i^2 for i = (1:n)[x[1]]], dims)
10-element DistributedArrays.DArray{Any,1,Array{Any,1}}:
1
4
9
16
25
36
49
64
81
100
julia>
Upvotes: 14
Reputation: 530
Hi Kira,
I am new on Julia, but facing the same problem. Try this approach and see if it fits your needs.
function f(x)
return x^2;
end
y=@parallel vcat for i= 1:100
f(i);
end;
println(y)
Regards, RN
Upvotes: 3