CODEWITHSUNDEEP
ruby-on-railsrubymultithreading
dc10
dc10

Reputation: 2198

How to implement a multithreaded infinite-task with subthreads in Rails

Inside my Rails 4 application I need to make API calls to a webservice where I can ask for an instance of stock quotes. I can ask for 1000 stocks in one request and four requests at a time which is the throttle limit.

The workflow goes like this: A user makes a request to my app and wants to get quotes for 12000 stocks.

So I chunk it into twelve requests and put them in a Queue. At application start I start a thread in a loop which is supposed to look at the queue and since I am allowed to make concurrent requests I'd like to make 4 requests in parallel.

I get stuck in several ways. First of all I need to take into consideration that I can get multiple requests of 12000 stocks at a time since different users can trigger the same request.

Second, I ll use the Thin web server wjich is multithreaded. So I guess I have to use a Mutex.

How can this be achieved?

Upvotes: 0

Views: 206

Answers (1)

Chris Heald
Chris Heald

Reputation: 62648

Queues are already threadsafe data structures, so you don't need a mutex to work with them.

You'd just start 4 threads at the start of your app, which each poll the queue for work, do some work, and then do something (which is up to you) to notify the DB and/or user that the work is complete. The actual workers will be something like:

work_queue = Queue.new
4.times do 
  Thread.new do
    loop do
      job = work_queue.pop
      # Do some work with the job
    end
  end
end

Queue#pop blocks until data is available, so you can just fire off those queues and the first thread waiting for data will get the job when it's pushed in, the next thread will get the next job, and so on. While there are no worker threads available, jobs will accumulate in the queue.

What you actually do with the output of the job is probably the more interesting question here, but you haven't defined what should happen when the quotes are retrieved.

You might also look at Sidekiq.

Upvotes: 1

Related Questions