CODEWITHSUNDEEP
ruby-on-railsruby-on-rails-4rails-activerecorddelayed-job
Vlad
Vlad

Reputation: 8268

How to insert thousands of records using delayed_job without timing out?

I need to insert thousands of records to a table on certain request. Right now this is how I do it: (Let's say I'm trying to pour same coffee to thousands of cups)

@cups.each do |cup|
  cup.delay.pour @coffee
end

Now normally this is not much of a problem, but when there are tens of thousands of cups, even though each task gets taken care of in the background, adding tasks itself takes a while. So what I've noticed is it starts adding several hundred tasks and then the request times out and the rest don't get added to the job queue.

I'm using unicorn and so far I'm dealing with this by increasing the allowed timeout ridiculously (like 200sec) and it is working kind of. But I can't be doing this forever. Anyone have any tips on how to deal with this properly? Thanks

Upvotes: 0

Views: 245

Answers (1)

Md Sirajus Salayhin
Md Sirajus Salayhin

Reputation: 5144

Try to use Active record mass insert.

CONN = ActiveRecord::Base.connection
TIMES = 10000

inserts = []
TIMES.times do
    inserts.push "(3.0, '2009-01-23 20:21:13', 2, 1)"
end
sql = "INSERT INTO user_node_scores (`score`, `updated_at`, `node_id`, `user_id`) VALUES #{inserts.join(", ")}"
CONN.execute sql

Upvotes: 1

Related Questions