CODEWITHSUNDEEP
ruby-on-railsbackground-processresquesidekiq
Avishai
Avishai

Reputation: 4722

Long-running Sidekiq jobs keep dying

I'm using the sidekiq gem to process background jobs in Rails. For some reason, the job just hang after a while -- the process either becomes unresponsive, showing up on top but not much else, or mysteriously vanishes, without errors (nothing is reported to airbrake.io).

Has anyone had experience with this?

Upvotes: 6

Views: 3769

Answers (2)

Henley Wing Chiu
Henley Wing Chiu

Reputation: 22515

I've experienced this, and haven't found a solution/root cause.

I couldn't resolve this cleanly, but came up with a hack.

I configured God to monitor my Sidekiq processes, and to restart them if a file changed.

I then setup a Cron Job that ran every 5 minutes that checked all the current Sidekiq workers for a queue. If a certain % of the workers had a start time of <= 5 minutes in the past, it meant those workers hung for some reason. If that happened, I touched a file, which made God restart Sidekiq. For me, 5 minutes was ideal but it depends on how long your jobs typically run.

This is the only way I could resolve hanging Sidekiq jobs without manually checking on them every hour, and restarting it myself.

Upvotes: 1

Mike Perham
Mike Perham

Reputation: 22208

Use the TTIN signal to get a backtrace of all threads in the process so you can figure out where the workers are stuck.

https://github.com/mperham/sidekiq/wiki/Signals

Upvotes: 8

Related Questions