CODEWITHSUNDEEP
laravelqueue
TKoL
TKoL

Reputation: 13912

Using multiple variable concurrent Laravel Queues

My use case:

My application will have any number of Users, and each of those Users will have various Jobs they want to process. These jobs should be able to happen concurrently -- some of them, anyway.

So, for example, if User1 sends through a command like UpdateStock and CompleteSale, and User2 sends through a command like UpdateStock, each of those commands needs to be queued up and allowed to run concurrently with each other.

I've been looking into Laravel Queues and it looks like, if I were to put all 3 jobs into one queue, they would only run one at a time, but if I put them into 3 different queues, they'd all be able to run concurrently. Do I understand that right?

So that brings me to my next issue then: I'll need one queue, per user, per command. But of course new users are signing up all the time, and if I need to run php artisan queue:listen --queue=user1:command1 & php artisan queue:listen --queue=user1:command2 & php artisan queue:listen --queue=user2:command1 ... to get this to work then... how is that even possible? When a new user could be added any minute...

I mean I could write some php that executes php artisan queue:listen --queue=userx:commandx programmatically after every time I add something to the queue, but I'm not sure that makes sense. What if that command has already run -- will it have 2 processes trying to process the same queue?

Perhaps I'm thinking about it the wrong way though. I'm feeling kinda lost about the whole thing. I could use some advice.

Upvotes: 4

Views: 2692

Answers (2)

Spholt
Spholt

Reputation: 4032

To answer the first point: Yes, If you have 3 Queues, each will process concurrently provided there are Queue workers listening to each of them.

As for the rest of the scenario, it sounds like another approach is needed as

one queue, per user, per command

sounds a little overkill!

In my experience, i would suggest having a handful of queues like "low", "default" & "high" priority queues with the higher priorit queues having more workers (and potentially a shorter wait time). You can configure these using programs like supervisor if you are running in a linux environment.

[program:high-priority-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --sleep=3 --tries=3 --queue=high
autostart=true
autorestart=true
user=user
startsecs=0
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/www/app/worker.log

[program:default-priority-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --sleep=3 --tries=3
autostart=true
autorestart=true
user=user
startsecs=0
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/www/app/worker.log

[program:low-priority-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --sleep=3 --tries=3 --queue=low
autostart=true
autorestart=true
user=user
startsecs=0
numprocs=2
redirect_stderr=true
stdout_logfile=/var/log/www/app/worker.log

If you are concerned about ensuring a sequence of jobs are excecuted in order, you may also want to look into Job Chaining (https://laravel.com/docs/6.x/queues#job-chaining)

Upvotes: 3

xyz
xyz

Reputation: 578

if I were to put all 3 jobs into one queue, they would only run one at a time, but if I put them into 3 different queues, they'd all be able to run concurrently. Do I understand that right?

Yes, the jobs are put after each other if you only have one queue.

What you are really talking about here is scaling.

The way I think most would go about this was to have sufficient queues. And then be able to scale that up. What you are proposing yourself (starting a queue for every new user) is also an option. Having multiple listeners for the same queue will give you problems. I did it by mistake some years ago, and the jobs were not probably "locked", meaning the same job would be executed multiple times.

Laravel Horizon brings you balancing options, meaning you could have new jobs go into free queues. So you do not have to dedicate one queue per user.

In case you need total concurrency (the jobs have to start at the exact same time), balancing in that sense is probably not enough.

Having multiple queues update the same stock numbers sounds a little risky though. I could think of some database locking issues and possible race conditions.

To give you a more specific recommendations I would need to understand why you need one queue per user in your case/example.

Upvotes: 1

Related Questions