CODEWITHSUNDEEP
ruby-on-rails-3ruby-on-rails-3.2sidekiq
RubyRedGrapefruit
RubyRedGrapefruit

Reputation: 12224

How can I run a continuous background job with Sidekiq?

I have been using Sidekiq successfully to run background jobs that are initiated by user actions in a Rails 3.2 application. My particular application involves sending and receiving data from and external source via a third party API.

I need to keep data synchronized with this external source by continuously checking for each user if there is data available to be downloaded.

All of my background jobs thus far are user-initiated as I mentioned above. My question is, if I want to continuously check for the existence of external data for each user, in a while loop I imagine:

# pseudocode
while true

  for user in User.active
    data = user.get_data
    next if data.blank?

    UserDataWorker.perform_async(user.id)

  end
end

then where do I put the code above? I get confused on how to spawn a continuous task. I don't want to put the code in a Sidekiq worker, because it will just be a job that never completes. Or should I do that?

I don't want to put it in a rake task, because then how do I monitor that rake task? Everything I've read about Sidekiq seems to point to a Worker running a task that is finite and is able to be completed or errored out. I appreciate any help you can offer.

Upvotes: 13

Views: 4078

Answers (2)

Jacka
Jacka

Reputation: 2560

I'd say use a gem like sidekiq scheduler: https://github.com/moove-it/sidekiq-scheduler.

You could start the work every minute or couple seconds easily. It would be running in the sidekiq process itself, which should work for you, since you need sidekiq running all the time.

Upvotes: 0

Utgarda
Utgarda

Reputation: 684

Ruby multithreading is just for that. Spawn a thread in your initializers folder:

Thread.new do

  loop do

    for user in User.active
      data = user.get_data
      next if data.blank?

      UserDataWorker.perform_async(user.id)
    end

    sleep 1

  end
end

If your checks don't need much resources, should be allright.

Upvotes: 9

Related Questions