CODEWITHSUNDEEP
rubymultithreadinglimit
Luke Chadwick
Luke Chadwick

Reputation: 1737

Process n items at a time (using threads)

I'm doing what a lot of people probably need to do, processing tasks that have a variable execution time. I have the following proof of concept code:

threads = []

(1...10000).each do |n|
  threads << Thread.new do
    run_for = rand(10)
    puts "Starting thread #{n}(#{run_for})"
    time=Time.new
    while 1 do
      if Time.new - time >= run_for then
          break
      else
          sleep 1
      end
    end
    puts "Ending thread #{n}(#{run_for})"
  end
  finished_threads = []
  while threads.size >= 10 do
    threads.each do |t|
      finished_threads << t unless t.alive?
    end
    finished_threads.each do |t|
      threads.delete(t)
    end
  end
end

It doesn't start a new thread until one of the previous threads has dropped off. Does anyone know a better, more elegant way of doing this?

Upvotes: 1

Views: 217

Answers (2)

Filipe Miguel Fonseca
Filipe Miguel Fonseca

Reputation: 6436

The work_queue gem is the easiest way to perform tasks asynchronously and concurrently in your application.

wq = WorkQueue.new 2 # Limit the maximum number of simultaneous worker threads

(1..10_000).each do
  wq.enqueue_b do
    # Task
  end
end

wq.join # All tasks are complete after this

Upvotes: 0

brianegge
brianegge

Reputation: 29892

I'd suggest creating a work pool. See http://snippets.dzone.com/posts/show/3276. Then submit all of your variable length work to the pool, and call join to wait for all the threads to complete.

Upvotes: 2

Related Questions