CODEWITHSUNDEEP
rubymultithreading
Sriharsha
Sriharsha

Reputation: 2443

using ruby threads with delay

I have written a ruby script which contains the list of files(~250) to be downloaded. Since sequential download will take ages. I thought of downloading each file in a separate thread. Even though I have given the delay of 20, it looks like all the threads are hitting the server at a time. Hence I am getting 502 error and none of the files are downloaded. How can I download all the files parallelly without overloading the server.

#list of fiiles
files = []
threads = []  
files.each do |file|
   threads  << Thread.new(file){ | file |
     sleep(20)  
     #Download the file using either curb or Net::HTTP
     sleep(20)
}
end

threads.each(&:join)

Upvotes: 0

Views: 575

Answers (3)

ll14m4n
ll14m4n

Reputation: 11

threads.each {|thread| thread.join; sleep 1}

Upvotes: -1

Kashyap
Kashyap

Reputation: 4796

Create a Thread-pool from which new threads are created. This way, you will not create more than the ones provided in the pool and avoid crashing the app/server. Alternatively, take a look at SideKiq This is a multi-threaded alternative to background queue processors like DelayedJob and Resque.

Upvotes: 0

ddb
ddb

Reputation: 1406

You should consider using doing this in the background.

Check out some of the gems that actually dont block and do the things in background - eg., DelayedJob. Also, check out some of the railscasts :

http://railscasts.com/episodes/128-starling-and-workling

http://railscasts.com/episodes/128-starling-and-workling

Upvotes: 0

Related Questions