CODEWITHSUNDEEP
rubyprocess
Uko
Uko

Reputation: 13386

Limit the number of forked processes

I'm trying to process an array and each element takes a reasonable time to process (≈30min). To speedup the process I want to use fork, so elements can be processes in parallel in separate processes. However my array has around 1000 elements, and I don't want to make 1000 forks at once.

I can do forks in batches of n and then wait until all of them finish with Process.waitall but is there a way where I can start a new process as soon a a previous one ended?

Upvotes: 4

Views: 975

Answers (1)

Uko
Uko

Reputation: 13386

The solution can be achieved in the following way:

# assign maximum number of processes
counter = @max_processes

# when a child process ends increase the counter of available processes
Signal.trap("CLD")  { counter += 1 }    

your_array.each do |element|

  # wait for a child process to end if there are no more "available processes"
  Process.wait if counter <= 0

  # before launching a process decrease the "available processes" counter
  counter -= 1

  fork {
    # do something with an element
  }

end

A more trivial option would be to split the array into sub-arrays and run them in parallel:

your_array.each_slice(@max_processes) do |elements|

  elements.each do |element|
    fork {
      # do something with an element
    }
  end

  Process.waitall

end

Upvotes: 6

Related Questions