CODEWITHSUNDEEP
ruby-on-railssidekiqruby-on-rails-5rails-activejob
Michael Curtis
Michael Curtis

Reputation: 3

Rails activejob with Sidekiq is spawning multiple jobs on perform_later

I have an fairly simple Rails 5 app for monitoring sites speeds. It sends jobs off to an external page speed testing service and periodically checks to see if the jobs are complete, and if so it calls and stores the data about the page.

Each project can have many pages, and each page can have many jobs, which is where the response from the testing service is stored.

There is an activejob, with a sidekiq backend, that is supposed to run every minute, check for pages that are due to run, and if any are found launch a job to enqueue it. It also checks if there are any enqueued jobs, and if they are found, it spools up a job to check the status and save data if it's complete. 

def perform() 
#Todo:- Add some way of setting the status into the view and check for active here - no way to stop jobs at the mo!
pagestorun = Page.where("runtime < ?", DateTime.now)

pagestorun.each do |page|
  job = Job.new(:status => "new")
  page.jobs << job
  updatedatetime(page)
end

 if pagestorun.count != 0
  #Run the queuejob task to pick out all unqueued jobs and send them to webpagespeedtest
  QueuejobsJob.perform_later
 end

#Check any jobs in the queue, then schedule the next task GetrunningtasksJob.perform_later FindjobstorunJob.set(wait: 1.minute).perform_later() end

This seems to work as expected for a while, but after 5 minutes or so two jobs seem to end up spawning at the same time. Eventually each of those spawn more of their own, and after a few days I end up with tens of thousands trying to run per hour. There's no errors or failing jobs as best I can tell, I can't find any reason why it'd be happening. Any help would be mundo appreciated :)

Upvotes: 0

Views: 1430

Answers (1)

bithavoc
bithavoc

Reputation: 1539

There's a chance the jobs are being retried due failures which, when overlapping with the regular 60 seconds schedule, could cause the double scheduling you're experiencing. For more info, see Error Handling in the Sidekiq wiki.

BTW I'm not entirely sure an active job it's the best way to run periodical tasks(unless your using sidekiq's enterprise periodical-jobs). Instead, I'd use a cron job running a rake task every 60 seconds, the rake task would schedule the jobs to check specific pages QueuejobsJob.perform_later page.id

Upvotes: 3

Related Questions