CODEWITHSUNDEEP
rubymultithreading
AJFaraday
AJFaraday

Reputation: 2450

Ruby repeat thread

So I've got a reporter method running in a separate thread, it looks a little like this

def report
  @reporters.each do |reporter|
    Thread.new{ reporter.report }
  end
end

The intention is that the reporter objects don't interfere with the performance of my main code loop. However this does create a lot of new threads, which seems to be taking up a lot of resources.

I'd like to instead collect those new threads and re-run them when the report method is called. Or send a single thread the report method on the reporter objects each time it's called.

Is there a way I can call methods in a separate thread after initialising it?

Upvotes: 0

Views: 123

Answers (3)

Myst
Myst

Reputation: 19221

Although I side with @AlexeyShein that a tested library is the better option, allow me to raise a few thoughts/issues as well as offer a basic thread-pool demo.

The code I offer is similar to what I wrote for the Iodine project, although simpler and a bit less optimized.

  1. Tasks in your thread pool might NOT complete:

    When the hosting OS/server shuts your application down, it might not wait for all your tasks to complete.

    The hosting OS/server will usually wait a short while (on Heroku it could be up to 30 seconds) before deciding your application had enough time and forcefully kills the process.

    This is normal and expected behavior.

  2. Tasks are NOT persistent:

    Whenever your application restarts, the task queue will be a fresh and clean queue.

    To resolve this issue, it is common to do two things: 1. Save the tasks in a database / module; and 2. run the background task manager in a different process (to avoid race conditions when scaling the application).

If you don't mind having incomplete tasks reinitiated after shutdown, here's a relatively simple thread-pool solution.

The following test code should show you how even small tasks might cause the rundown to linger after exit was called. If your application will linger for long enough, it could be forcefully shut down by your host.

class ThreadPool
    def initialize number_of_threads
        @queue = Queue.new
        @stop = false
        @threads = []
        number_of_threads.times do
            @threads << Thread.new { work until @stop }
        end
        Kernel.at_exit { rundown }
    end

    def run &block
        return false unless block
        @queue << block
        block
    end

    protected

    def work
        begin
            @queue.pop.call
        rescue => e
            warn "A background task failed with #{e.message}"
        end
    end

    def rundown
        sleep 0.1 until @queue.empty?
        stop
        join
        work until @queue.empty?
    end

    # will wait for all threads to finish
    # if stop isn't called, this will hang forever!
    def join
        @threads.each {|t| t.join rescue true }
    end

    # will stop the thread pool and let the threads exit normally
    def stop
        @stop = true
        (@threads.count * 2).times {  run { nil } }
    end
end



BK = ThreadPool.new 12
100.times {|i| BK.run {sleep 0.2; puts i.to_s} }

exit

Upvotes: 1

Aleksey Shein
Aleksey Shein

Reputation: 7482

Multithreading can be tricky, so I think the best option in your case would be to use some battle-tested library like sidekiq that would take all heavy-lifting on itself.

Upvotes: 0

sawa
sawa

Reputation: 168081

That design pattern is called thread pool. There is plenty of information on the internet regarding Ruby implementation of thread pool.

Upvotes: 1

Related Questions