CODEWITHSUNDEEP
ruby-on-railsrubymultithreading
Carl Wright
Carl Wright

Reputation: 55

Rails concurrent requests

So I have a rails app that parses a long json file - during which it sends off requests to an external API, which seem to take forever. My question is, is it as simple as to stick each request into a separate thread in order to speed it up? I don't see why I can't have say 3 or 4 threads running concurrently sending off requests ... the only issue I see is currently the method displays an errors page if there are any errors present at the end of parsing - and this possibly couldn't happen as the other threads might still be running? .. eventually I want to make this method into a rake task and run it using a cron job that renders a html page so that should be better correct?

def load_movies
 @errors = []
 movie_list = get_data

 movie_list.first(50).each do |movie_data|
 ------------ Thread.new do -----------
  movie = Movie.new
  movie.construct_movie movie_data

  if !movie.valid?
    @errors << movie
  else
   movie.save
  end
  ----------- end thread ------------
  unless @errors.count > 0
  redirect_to root_path
end
end

is it as simple as to do what is above and it will run off open multiple threads to send the requests (which can be assumed to be in construct_movie

Upvotes: 3

Views: 3354

Answers (1)

jrochkind
jrochkind

Reputation: 23317

You can do this, yes. I have done it, yes.

You can run into some gotchas with multi-threaded use of ActiveRecord, if you're using ActiveRecord. For doing multi-threaded stuff in ruby, I recommend the concurrent-ruby gem, which provides thread pools and higher level abstractions like Futures. https://github.com/ruby-concurrency/concurrent-ruby

You can do this, although, yeah, it can get tricky -- if a background job queue works, that might be a less reinventing-the-wheel solution for you. Whether a background job queue or threads, you're going to have to figure out how to display 'results' -- although with the straight threads approach, I guess your request can wait on all the threads and their results, before displaying results (yeah, I've done this too). That can cause it's own problems with having slow-running requests (you'll probably want to deploy with puma or passenger enterprise to avoid blocking your entire app process on them).

From my own experience doing similar things, I don't have an easy answer, you'll run into some challenges no matter what route you take, I think. If you can figure out a way to make a background job queue work, I think that will generally give you less gotchas than directly creating threads in the app process.

Upvotes: 3

Related Questions