CODEWITHSUNDEEP
parallel-processingjulia
warsaga
warsaga

Reputation: 273

Julia: Creating local variables on workers in parallel context

Using DistributedArrays in cases when the worker only needs to store unshared data seems overly complicated. I would like to do

r=remotecall(2,a=Float64[])
remotecall(2,setindex!,a,5,10) #Error

or

r=remotecall(2,zeros,10)
remotecall(2,setindex!,r,5,10) #Error.

I would like to do this for each worker and then access the array in an async context. Perform some computations and then fetch the results. I am not sure of this is possible because of the let behavior of async

Below I have made an simplified example for which I modified the pmap example form the docs. T

times=linspace(0.1,2.0,10) # times in secs representing different difficult computations
sort!(times,rev=true)

np = nprocs()
n = length(times)

#create local variables
for p=1:np
    if p != myid() || np == 1
        remotecall(p,stack = Float64p[]) #does not work
    end
end

@everywhere function fun(s)
    mid=myid()
    sleep(s)
    #s represents some computation save to local stack
    push!(stack,s)
end




#asynchronously do the computations
@everywhere i = 1
function nextidx()
    global i
    idx=i;
    i+=1;
    return idx;
end
@sync begin
    for p=1:np
        if p != myid() || np == 1
            @async begin
                j=1
                res=zeros(40);
                while true
                    idx = nextidx()
                    if idx > n
                        break
                    end
                    remotecall(fun, times[idx])
                end
            end
        end
    end
end

# collect the results of the computations
for p=1:np
    if p != myid() || np == 1
        tmpStack=fetch(p,stack)
        #do someting with the results
    end
end

Upvotes: 1

Views: 1217

Answers (1)

sjnahn
sjnahn

Reputation: 63

By using 'global' when you modify the global variable of the worker (e.g., set by @everywhere a = 3), you may be able to resolve your problem. Check out the example code below. 

@everywhere a = 0
remotecall_fetch(2, ()->a)  # print 0

@everywhere function change_a(b)
    global a
    a = b
end

b = 10
remotecall_fetch(2, change_a, b) 
remotecall_fetch(2, ()->a)  # print 10

Upvotes: 2

Related Questions