CODEWITHSUNDEEP
parallel-processingidl-programming-language
AstroCab
AstroCab

Reputation: 11

IDL:How to divide FOR loop in N parts for parallel execution?

I have a time consuming loop of length 300. I would like to execute in parallel.

Pseudocode:

for t=0, 300 do begin
output_data[t] = function(input_data(t))
endfor

• The function() for each iteration is completely the same • The input_data(t)) is stored in a file

Is possible divide the 300 iterations in to K parallel processes (where k is the CPU number)? I found split_fot.pro but if I understand correctly it is for divide the different processes in the same nth cicle of loop. How can I do?

Thank you!!

Upvotes: 0

Views: 1261

Answers (2)

hgh
hgh

Reputation: 11

You can use IDL_IDLBridge along with object arrays to create multiple child processes.

Child processes do not inherit variables from main process, so you may want to use SHMMAP, SHMVAR, SHMUNMAP to share variables across child processes, or use SETVAR method of IDL_IDLBridge if memory is not a problem.

As an example, below I create 5 child processes to distribute the for-loop:

dim_input = size(input_data, /dim) ; obtain the data dimension
dim_output = size(output_data, /dim)
shmmap, dimension=dim_input, get_name=seg_input   ; set the memory segment
shmmap, dimension=dim_output, get_name=seg_output 
shared_input = shmvar(seg_input)
shared_output = shmvar(seg_output)
shared_input[0] = input_data ; assign data to the shared variable
                             ; shared_data[0, 0] if data is 2d
shared_output[0] = output_data

; initialize child processes
procs = objarr(5)
for i = 0, 4 do begin
    procs[i] = IDL_IDLBridge(output='')
    procs[i].setvar, 'istart', i*60
    procs[i].setvar, 'iend', (i+1)*60 - 1
    procs[i].setvar, 'seg_input', seg_input
    procs[i].setvar, 'seg_output', seg_output
    procs[i].setvar, 'dim_input', dim_input
    procs[i].setvar, 'dim_output', dim_output
    procs[i].execute, 'shmmap, seg_input, dimension=dim_input'
    procs[i].execute, 'shmmap, seg_output, dimension=dim_output'
    procs[i].execute, 'shared_input = shmvar(seg_input)'
    procs[i].execute, 'shared_output = shmvar(seg_output)'
endfor

; execute the for-loop asynchronously
for i = 0, 4 do begin
    procs[i].execute, 'for t=istart, iend do ' + $
                      'shared_output[t] = function(shared_input[t])', $
                      /nowait
endfor

; wait until all child processes are idle
repeat begin
    n_idle = 0
    for i = 0, 4 do begin
    case procs[i].status() of
        0: n_idle++
        2: n_idle++
        else:
    endcase
  endfor
  wait, 1
endrep until (n_idle eq 5)

; cleanup child processes
for i = 0, 4 do begin
    procs[i].cleanup
    obj_destroy, procs[i]
endfor

; assign output values back to output_data
; unmap the shared variable
output_data = shared_output[0]
shmunmap, seg_input
shmunmap, seg_output
shared_input = 0
shared_output = 0

You may also want to optimize your function for multi-processing. Lastly, to prevent multiple accesses to the memory segment, you can use SEM_CREATE, SEM_LOCK, SEM_RELEASE, SEM_DELETE functions/procedures provided by IDL.

Upvotes: 0

mgalloy
mgalloy

Reputation: 2386

I have some routines in my library that you could use to do something like the following:

pool = obj_new('MG_Pool', n_processes=k)
x = indgen(300)
output_data = pool->map('my_function', x)

Here, my_function would need to accept an argument i, get the data associated with index I, and apply function to it. The result would then be put into output_data[i].

You can specify the number of processes you want to use for the pool object with the N_PROCESSES keyword or it will just automatically use the number of cores you have available.

The code is in my library, check the src/multiprocessing directory. See the examples/multiprocessing directory for some examples of using it.

Upvotes: 1

Related Questions