CODEWITHSUNDEEP
rubymultithreading
DrPppr242
DrPppr242

Reputation: 1951

Limiting concurrent threads

I'm using threads in a program that uploads files over sftp. The number of files that could be upload can potentially be very large or very small. I'd like to be able to have 5 or less simultaneous uploads, and if there's more have them wait. My understanding is usually a conditional variable would be used for this, but it looks to me like that would only allow for 1 thread at a time.

cv = ConditionVariable.new

t2 = Thread.new {
  mutex.synchronize {
    cv.wait(mutex)
    upload(file)
    cv.signal
  }
}

I think that should tell it to wait for the cv to be available the release it when done. My question is how can I do this allowing more than 1 at a time while still limiting the number?

edit: I'm using Ruby 1.8.7 on Windows from the 1 click installer

Upvotes: 0

Views: 1347

Answers (2)

Victor Moroz
Victor Moroz

Reputation: 9225

A word of caution -- there is no real concurrency in Ruby unless you are using JRuby. Also, exception in thread will freeze main loop unless you are in debug mode.

require "thread"

POOL_SIZE = 5

items_to_process = (0..100).to_a

message_queue = Queue.new

start_thread = 
  lambda do
    Thread.new(items_to_process.shift) do |i|
      puts "Processing #{i}"
      message_queue.push(:done)
    end
  end

items_left = items_to_process.length

[items_left, POOL_SIZE].min.times do
  start_thread[]
end

while items_left > 0 
  message_queue.pop
  items_left -= 1
  start_thread[] unless items_left < POOL_SIZE
end

Upvotes: 1

coreyward
coreyward

Reputation: 80128

Use a ThreadPool instead. See Deadlock in ThreadPool (the accepted answer, specifically).

Upvotes: 2

Related Questions