CODEWITHSUNDEEP
phpmultithreadingmultiprocessingamqpsupervisord
davidaam
davidaam

Reputation: 449

Multi-threading RabbitMQ consumers with PHP

I'm developing an app which uses rabbitmq for queuing tasks. I have a worker script which listen to the queue and do the job. What I'm wondering is how to manage multiple consumers in parallel. I initially thought of using pthreads for multi-threading, but ran into all kind of weird issues, and for what I've read in its Github page, it's not possible, something about AMQP extension not being thread safe.

I also read something about Supervisord (haven't really try it though) but what I read suggested me I can't just change dynamically modify the numprocs without restarting the other processes. This would actually be on demand, having some backend and when some variables are modified, the number of workers is modified.

Now, my best shot is to run multiple processes of the worker script. But how to manage them? One of the main issues is that the number of workers needed may vary over time, and how to ensure that if the server crashes, they will be up and running asap.

Is there some sort of recommended pattern for doing this? I thought of running a script with upstart or maybe a cron, which forks the processes, but then when it run again it would have to check if the number of worker's available suffice, and to be able to do that I think I would have to store the PIDs in a database. I guess there may be another way.

UPDATE: I thought of a way to use Supervisord and modifying the numprocs in order to run new workers when needed (this would be actually rare but need to cover the scenario). It would work this way: Whenever I need to run new workers, I would send a message to the current workers to stop listening to the work queue, when they are done doing their current tasks, I restart the workers, including the new ones.

What do you think about it? I know it's probably not the best design, but it's the best I can do. I have some doubts on how to implement the part of sending the message with rabbitmq (first time working with a message broker) but I think I can handle it

Upvotes: 1

Views: 2946

Answers (1)

pinepain
pinepain

Reputation: 12869

Yes, you can use multiple processes (php scripts) controlled by supervisord (or monit) or you may utilize threading, but in this case you have to open new connection (not only channel) per each thread to prevent various errors (with both php-amqp extension and php-amqplib library).

Keep in mind that PHP was not designed to manage resources in threaded environment, so multiprocess way looks more stable and maintainable than multithreaded.

Upvotes: 2

Related Questions