CODEWITHSUNDEEP
ruby-on-railsrubysidekiq
Daniel Costa
Daniel Costa

Reputation: 295

How to queue a non-existing worker

I'm trying to queue a Sidekiq worker that does not exist in my project but exists in some other project sharing the same redis server.

In our current situation, we have two servers:

Side note: In the beginning, our API was doing all the work but as some workers started taking too much memory, we switch to the processing server.

When we need to execute a job on the processing server we have these actions happening:

  1. API calls POST processing.com/some_job

  2. Processing queues SomeJobWorker on queue processing

  3. Processing executes SomeJobWorker from queue processing

  4. Processing calls POST api.com/webhooks/some_job_result

  5. API queues SomeJobResultWorker on queue api

  6. API executes SomeJobResultWorker from queue processing

This got me thinking if I could somehow, from the API server directly queue SomeJobWorker then I could get rid of the REST API on our processing server.

... then I could get rid of the webhooks endpoint on the API server and do the same from the processing server

I would end up with something like this:

  1. API calls queues SomeJobWorker

  2. Processing executes SomeJobWorker

  3. Processing queues SomeJobResultWorker

  4. API executes SomeJobResultWorker

Is there a way for me to queue a worker that does not exist on my local code base?

Upvotes: 0

Views: 296

Answers (1)

Mike Perham
Mike Perham

Reputation: 22238

You have two shards. Each Redis is a shard. Read this page:

https://github.com/mperham/sidekiq/wiki/Sharding

You can push a job using the class name:

client = Sidekiq::Client.new(PROCESSING_REDIS)
client.push('queue' => 'api', 'class' => 'FooWorker', 'args' => [1,2,"foo"])

Upvotes: 1

Related Questions