CODEWITHSUNDEEP
phppythonmessage-queue
Farzher
Farzher

Reputation: 14563

PHP push to message queue; process with multithreaded python client?

Here's what I'm trying to do: use PHP to push a message into a queue (either Beanstalkd, IronMQ, Amazon SQS).

Then I want a Python script to instantly pick up the message. I guess I'll just have 1 Python script running a while(true) loop to keep polling the message server?

Then process it using a new thread for each job. If there's 10 messages in the queue, I want python to run 10 threads, 1 for each job.

My questions are:

  1. Is this a solid way of doing things, or is there a better way to set this up?
  2. How do I ensure my Python script is up and polling forever?

Upvotes: 0

Views: 525

Answers (1)

Aya
Aya

Reputation: 41950

Is this a solid way of doing things, or is there a better way to set this up?

Sounds reasonable to me, although if you're expecting a large number of simultaneous jobs, you may want to limit the total number of threads by using a thread pool.

You won't gain much more CPU performance, for CPU-intensive threads, when the total number of threads exceeds the total number of CPU cores, and if the threads require significant disk I/O, you'll want to limit them to avoid thrashing the disks.

How do I ensure my Python script is up and polling forever?

It's common for daemon processes to have another process which monitors them, and restarts them if they crash or become unresponsive. A simple cronjob might suffice for this purpose.

Upvotes: 1

Related Questions