CODEWITHSUNDEEP
pythonmultithreadingshellsubprocess
user624208
user624208

Reputation: 49

Threading subprocesses in Python

If I use Python's Threading library can I accomplish a batch of subprocess processes faster? Let's say for instance I need to convert 100 .wav files to .mp3 files. If I wrap 'ffmpeg' in a Python script that used threading could I accomplish the task much faster? Does threading enable me to actually use all 8 threads available in my i7?

I recently stumbled across a Python script that pings a list of hosts but utilizes threading to make things faster. The script seemed succinct and was easy enough for a beginner such as myself to read and understand. So this is where my other question arises: What is all the talk about running parallel tasks being so complex? Maybe I don't understand parallel and parallel != threading. If it is that simple then why wouldn't people use threading for any and all batch conversions when running on a modern processor?

Upvotes: 4

Views: 6411

Answers (3)

dolphin
dolphin

Reputation: 1233

Here is a simple and working solution for your case. I use it often and it proves to be very useful! This code creates as many threads as cores and lets them execute a (large) number of tasks (in this case, calling a shell program):

import Queue
import threading
import multiprocessing
import subprocess

q = Queue.Queue()
for i in range(30): #put 30 tasks in the queue
    q.put(i)

def worker():
    while True:
        item = q.get()
        #execute a task: call a shell program and wait until it completes
        subprocess.call("echo "+str(item), shell=True) 
        q.task_done()

cpus=multiprocessing.cpu_count() #detect number of cores
print("Creating %d threads" % cpus)
for i in range(cpus):
     t = threading.Thread(target=worker)
     t.daemon = True
     t.start()

q.join() #block until all tasks are done

Upvotes: 4

BrainStorm
BrainStorm

Reputation: 2046

making your program multithreaded with threading will certainly make it faster if used correctly, but you can't really control how much of your processor, or how many cores your app will use because it's all handled by the OS, the multiprocessing module in python is another approach you could take, but it would be better if you read about this subject with greater detail before trying out these modules, I recommend this tutorial.

Upvotes: 2

Kracekumar
Kracekumar

Reputation: 20419

You should use multiprocessing module in python, because you can make use more than one core in your computer.Try this multiprocess module in python

Upvotes: 0

Related Questions