Laravel - how to move jobs form one redis queue to other redis queue?

Question

For example I have two queues "high" and "low". I have 1000 jobs in "high" queue and 0 in "low" queue. Now I want to move for example 500 jobs from "high" queue to "low" queue.

For start it would be fine to find out how to move all jobs not only half of them.

I can get all jobs with this command :

\Redis::lrange('queues:' . $name, 0, -1);

But how to move them, any idea ?

Answer 1

To sum up the dicussion in the comments, here some recommendations and additional information.

Manually adjusting the Redis queue

It is not recommended to intervene with the redis queue manually. Do not alter the queue by hand. Instead, let the queue workers handle the queue that has an unexpected (high) load of work. You can also spawn additional queue workers temporarily to get the work done faster.

Maybe take the unbalanced queue loads into account when working on future features though.

Fixing the queue work load

To fix the queue work load, there are a number of solutions. What they have in common is that we share resources between the individual queues. The only difference is the way how this is achieved.

For the following options, I'll use a very basic example. Imagine a simple cloud application where users can buy some computation power (for whatever). To make things more interesting, the users of the application can also buy a priority queue ticket, which guarantees them priority processing. In other words, their requests should (but don't have to) be processed with priority.

1. Rescaling the queue worker processes

One way to share resources is to up- and down-scale queue worker processes based on the workload. This means we reduce the queue worker processes for one queue so that we have the resources for additional queue workers for the other queue.

In our example, we would probably expect our users to use 9 out of 10 times the non-priority processing because they do not want to pay the extra for faster processing. This means we would normally have 9 work items in the low priority queue for 1 work item in the high priority queue. So that the priority processing makes sense, we would now need something like 3 queue worker processes per queue. We would start the processes like this:

• 3x php artisan queue:work --queue=high
• 3x php artisan queue:work --queue=low

If now the high priority queue has suddenly a lot more work items (e.g. caused by a sale of the priority queue ticket), we would need to rescale our queue workers accordingly. To do so, we would have to manually kill some of the --queue=low worker processes and start more of the --queue=high workers.

Because this is quite cumbersome to do by hand (and we devs also need to sleep once in a while), there is a solution to this which is called Laravel Horizon. When configured properly balance mode set to auto, Horizon will make sure that queues with higher work loads do get more attention than queues with less work load. In other words, Horizon will try to achieve equal waiting times across queues.

2. Let queue workers work on multiple queues

A less complex approach is to let queue workers listen and work on multiple queues. This can be done by passing multiple, comma-separated queues to the --queue parameter: --queue=high,low

When doing so, we instruct the queue worker to handle work items on the high queue with priority over the work items on the low queue. That means the worker will always clear the high queue entirely before taking work from the low queue. If, after processing a job on the low queue, the worker finds an item on the high queue, he will jump back to this queue. So each time the worker looks for a new work item, he will first look on the high queue and if there are no work items, he will go to the low queue.

For our example above, we could for example deploy the following 6 queue workers:

• 4x php artisan queue:work --queue=high,low
• 2x php artisan queue:work --queue=low

In this case we would have (proportionally) a lot more worker resources for the high queue than for the low queue. But the resources would help out the other works of the low queue when they do not have any priority work to do.

We could of course also add the high queue as fallback to our low queue workers:

• 4x php artisan queue:work --queue=high,low
• 2x php artisan queue:work --queue=low,high

This way they would be able to help our high priority queue workers in case of work overloads like in the sale situation described above.

Conclusion

Simple solutions are often better. When your work loads do not explode all of a sudden and you can predict them quite good, then go for the second approach (but make use of supervisor so that you don't have to manually start the workers all the time and also to make sure they restart in case of a crash).

If, on the other hand, your work loads vary a lot and you cannot really tell in advance how many queue workers you need, Horizon may be worth a look. Personally, I also use Horizon for rather simple projects because it is easy to set up and it takes away work of me where I have to think a lot about loads and stuff.