CODEWITHSUNDEEP
phppythonrubymultithreading
jahmax
jahmax

Reputation: 8609

How to do this (PHP) in python or ruby?

My app takes a loooong list of urls, and split it in X (where X = $threads) so then I can start a thread.php and calculate the urls for it. Then it does GET and POST request to retrieve data

I am using this:

for($x=1;$x<=$threads;$x++){
    $pid[] = exec("/path/bin/php thread.php <options> > /dev/null & echo \$!");
}

For "threading" (I know its not really threading, is it forking or what?), I save the pids into a file for later checking if N thread is running and to stop them.

Now I want to move out from php, I was thinking about using python because I'd like to learn more about it.

How can I achieve this kind of "threading" with python? (or ruby)

Or is there a better way to launch multiple background threads in python or ruby that runs in parallel (at the same time)?

The threads doesn't need to communicate between each other or with a main thread, they are independent, they do http request and interact with a mysql db, they may need to access/modify the same table entries (I haven't tought about this or how I will solve it yet).

The app works with "projects", each project has a "max threads" variable and I use a web interface to control it (so I could still use php for the interface [starting/stopping threads] in the new app).

I wanted to use 

from threading import Thread

in python, but I've been told those threads wont run in parallel but once at a time.

The app is intended to run on linux web servers.

Any suggestion will be appreciated.

Upvotes: 1

Views: 326

Answers (2)

ars
ars

Reputation: 123508

For Python 2.6+, consider the multiprocessing module:

multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows

For Python 2.5, the same functionality is available via pyprocessing.

In addition to the example at the links above, here are some additional links to get you started:

Upvotes: 1

Charles
Charles

Reputation: 51411

You don't want threading. You want a work queue like Gearman that you can send jobs to asynchronously.

It's worth noting that this is a cross-platform, cross-language solution. There are bindings for many languages (including Python and PHP) provided officially, and many more unofficially with a bit of work with Google.

The original intent is effectively load balancing, but it works just as well with only one machine. Basically, you can create one or more Workers that listen for Jobs. You can control the number of Workers and the types of Jobs they can listen for.

If you insert five Jobs into the queue at the same time, and there happen to be five Workers waiting, each Worker will be handed one of the Jobs. If there are more Jobs than Workers, the Jobs get handled sequentially. Your Client (the thing that submits Jobs) can either wait for all of the Jobs it's created to complete, or it can simply place them in the queue and continue on.

Upvotes: 1

Related Questions