CODEWITHSUNDEEP
rubymultithreadingcallbackeventmachine
everm1nd
everm1nd

Reputation: 731

Ruby: EventMachine and separate callback queues

EventMachine executes all operations in separate threads and all callbacks of this operations in one mainthread. It's very useful when you need reduce competitiveness in data processing between portions of data. You can recieve data from network concurent and process results in queue. But I need go even futher. I have many users on my site and want to make separate queue for each of them. So it should behave like single EM for each user. But I can't. EventMachine is singletone.

Thread.new do
  EM.run
end

def request_from_user
  operation = Proc.new {
    # network request
  }
  callback = Proc.new {
    # some heavy data processing
    2*2
  }
  5.times do
    EM.defer(operation, callback)
  end
end

3.times do
  request_from_user
end

So we have callback-mess in mainthread queue. Some data will be processed in end of queue, some in the middle and it's awful. Heavy load will enlarge the queue and some users will recieve results with very big delay. If I run request_from_user in single thread callbacks will be executed in one EM main thread anyway.

So what can I do with it? Is it possible to create separate callbacks queue for each user and execute it concurent or not? May be even without EventMachine. Celluloid, whatever...

Upvotes: 0

Views: 630

Answers (2)

dj2
dj2

Reputation: 9618

Why not just move the work in callback either to operation (so, callback only fires when you're done processing the data), or to an EM.defer call? You don't want to be doing your heavy data processing on the main thread as that's going to cause all kinds of latency.

Upvotes: 0

rogerdpack
rogerdpack

Reputation: 66741

If you can listen on separate ports, then you could have one server/port per user. If not Ruby does have a Queue object that may be of some use to you: http://www.ruby-doc.org/stdlib-1.9.3/libdoc/thread/rdoc/Queue.html for farming processes out to other threads.

Upvotes: 1

Related Questions