CODEWITHSUNDEEP
phplaravel
Wern Ancheta
Wern Ancheta

Reputation: 23297

Multiple queue workers in Laravel

I've specified numprocs as 8 in the supervisor configuration. But everytime I dispatch a job, it basically processes the same job 8 times:

[2017-08-14 02:00:11] Processed: App\Jobs\UpdateOrders
[2017-08-14 03:00:09] Processed: App\Jobs\UpdateOrders
[2017-08-14 04:00:07] Processed: App\Jobs\UpdateOrders
[2017-08-14 05:00:06] Processed: App\Jobs\UpdateOrders
[2017-08-14 06:00:10] Processed: App\Jobs\UpdateOrders
[2017-08-14 07:00:08] Processed: App\Jobs\UpdateOrders
[2017-08-14 08:00:09] Processed: App\Jobs\UpdateOrders
[2017-08-14 09:06:39] Processed: App\Jobs\UpdateOrders

Here's what my supervisor config looks like:

[program:product-manager]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:listen
numprocs=8
autostart=true
autorestart=true
directory=/var/www/html
redirect_stderr=true
stdout_logfile=/var/www/logs/product-manager.log

How do I set it so that it only uses one worker if I've only dispatched one job. Or utilize all 8 workers if I dispatched 8 jobs. Is that the way it works? Am I having a wrong idea about how multiple workers in laravel queue works? All I really want is to have each worker work on a separate job that I dispatched.

Upvotes: 2

Views: 5410

Answers (1)

wally
wally

Reputation: 3602

I've asked if the OP can be updated with a clarification as to the queue configuration. This answer may be proven wrong subject to that update.

Your supervisor configuration is spawning multiple workers (using the Laravel built-in worker, which isn't super scalable or necessarily "production ready", but fine for development and testing) who are all trying to consume from a queue (which I'm guessing is probably a database table).

It's the queue's responsibility (for example Amazon SQS or AMQP etc) to ensure that a message on the queue is only delivered (consumed) once. I expect you're using a queue which doesn't enforce that (like the database), and multiple workers are consuming the same message together. (A kind of race condition.)

Possibly my understanding of how the database and artisan consumer works is wrong - in which case I trust I'll be down voted. That'd be where I'd start looking though.

Upvotes: 1

Related Questions