CODEWITHSUNDEEP
ruby-on-rails-3delayed-job
Evgeniya Manolova
Evgeniya Manolova

Reputation: 2650

delayed_job: how to force processing of failed jobs

I am maintaining a rails app, on which i'm running the delayed_job gem for sending emails.

I just noticed that all my delayed jobs've been failing the last few days due to a bug in the application. Now the bug is fixed and I want to process the jobs asap, but they are with already too many failed attempts, and the worker pulls them from the database with a huge delay.

I've tried by updating the delayed_jobs table and set the number of attempts to a smaller number and the run_at attribute to the current time, but still didn't help.

Can you tell me how can I force the worker to execute them ?

Upvotes: 4

Views: 9572

Answers (4)

Bret Weinraub
Bret Weinraub

Reputation: 2283

None of the existing answers has this exactly right, so I add this here.

The magic is to convince delayed jobs that these jobs really aren't failed, so via the rails db console:

-> rails db
development# update delayed_jobs set run_at = now() - interval '3 hours', attempts = 0, failed_at = null;
UPDATE 30
development# \q

-> rake jobs:workoff  # a good one to use, because it will return immediately if no jobs are found

Either a "failed_at" or attempts field can keep a job from running.

Upvotes: 3

Evgeniya Manolova
Evgeniya Manolova

Reputation: 2650

Ok, so finally i got it!

The trick was really to update the run_at attribute to the current time, but the current for the app - which was 3hours behind the database.

When I set it to now() - interval '3 hours' all jobs were processed.

edit:

@rodzyn, I've tried your suggestion, but still couldn't get it work:

[20] pry(main)> Delayed::Job.all.size
  Delayed::Backend::ActiveRecord::Job Load (0.6ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" 
=> 1
[21] pry(main)> Delayed::Job.first.invoke_job
  Delayed::Backend::ActiveRecord::Job Load (0.5ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" LIMIT 1
  Order Load (0.4ms)  SELECT "orders".* FROM "orders" WHERE "orders"."id" = $1 LIMIT 1  [["id", "328"]]
  User Load (0.5ms)  SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1
=> nil
[22] pry(main)> Delayed::Job.all.size
  Delayed::Backend::ActiveRecord::Job Load (0.6ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" 
=> 1
[23] pry(main)>

Upvotes: 3

Syed Aslam
Syed Aslam

Reputation: 8807

Now the bug is fixed and I want to process the jobs asap, but they are with already too many failed attempts, and the worker pulls them from the database with a huge delay.

That means, and since the jobs are not deleted from the table, there are still some more tries left for delayed job. The default behavior of delayed job is to read 5 jobs from the queue when finding an available job. One way without much of the code change is to have a setting in delayed config which would pick up more jobs from the queue and execute. You can configure this by setting Delayed::Worker.read_ahead.

# config/initializers/delayed_job_config.rb
Delayed::Worker.destroy_failed_jobs = false
Delayed::Worker.read_ahead = 10

Delayed::Worker.destroy_failed_jobs to avoid deleting of jobs after maximum attempts, which is also a configurable item.

Delayed job checks for available jobs every 5 seconds in the database and it would try a particular job after 5 + N ** 4 seconds. So, say if a particular job has failed 24 times already, then its turn would arrive after 331781 seconds, ie., roughly after 3 days if I am not wrong.

Upvotes: 1

Marcin Olichwirowicz
Marcin Olichwirowicz

Reputation: 890

You can start it manually, try

Delayed::Job.all.each { |j| j.invoke_job }

or

Delayed::Job.all.each { |j| j.payload_object.perform }

Upvotes: 10

Related Questions