CODEWITHSUNDEEP
laravelqueueworkerlumen
eComEvo
eComEvo

Reputation: 12589

Dynamically created worker daemon queues in Laravel/Lumen

I need to create multiple dedicated daemon worker queues per type of job on a per account basis.

For instance, suppose I have 1000 accounts that needs a dedicated listener for each of it's job types. Each job type needs it's own dedicated worker to process 50 instances in parallel. I don't want any account to block the workers from another account and I want all workers for an account to run asynchronously. I want all 50,000 workers humming along happily without blocking.

How could I dynamically spawn listeners in-code for each account?

Could I expect to have a heavy memory footprint from running 1000's of daemon listeners or are they lightweight?

Is a particular service such as Redis, Beanstalkd, etc better suited to such goals?

Planning to try this in Laravel or Lumen 5.2.

Upvotes: 2

Views: 1183

Answers (1)

Mysteryos
Mysteryos

Reputation: 5791

Based on a linux VM with 2Gb of RAM & CPU Frequency of 3.3GHz (1 Core), and the stat result from ps aux command:

Ps aux stats

The Resident Set Size for one queue is at its lowest ~52 MB (50888 KiB) and highest ~70 MB (68752 KiB).

1000 Workers = 52,000 MB (52 GB) - 70,000 MB (70 GB)

The cpu usage seems rather insignificant.

Note: my php-fpm is currently on version 5.x. If you target php 7.x, more performance gains can be attained.

Upvotes: 1

Related Questions