CODEWITHSUNDEEP
pythonpipesubprocess
Frank Yang
Frank Yang

Reputation: 49

Python - How to do non-blocking read from PIPE in subprocess.Popen?

I've posted similar question last week and this post reflects my trial and the problem I am facing now.

The program to invoke through Popen is a command line program. I use one thread to read one item from queue and send it to the stdin and get the response from stdout. However it hangs in the proc.stdout.read(). I did see it works fine with expected output last Friday and then when I did some changes today, it hangs. The change I made is replace read() with readlines() and use a loop to iterate the results. I am aware that readlines() would probably block but when I reversed the code to where it was with read() last Friday, it blocks, too. I completely get lost to it now. Any possible causes?

Below is the code to get one sentence from queue and feed it to the java program to get a response:

''' below is the code for worker thread. '''
def readQueue(proc, queue):
    print 'enter queueThread.\n'
    global notEmpty
    notEmpty = True
    while notEmpty:
        try:
            sen = queue.get()
            proc.stdin.write(sen.strip())
            res = proc.stdout.read()
            print res.strip(), ' ', sen
            queue.task_done()
        except Empty:
            break
    print 'leave queueThread.'

The main thread below is to read each line from a file and put it in a queue for the worker thread to process item by item:

def testSubprocess():
    ee = open('sentences.txt', 'r')
    #ff = open('result.txt', 'w')   # print it to stdout first before really write to a file. 
    lines = ee.readlines()
    cmd = ['java', 
           '-cp', 'someUsefulTools.jar', 
           'className', 
           '-stdin',]   # take input from stdin
    proc = Popen(cmd, stdout=PIPE, stdin=PIPE, stderr=PIPE, bufsize=-1, universal_newlines=True)
    q = Queue()
    for sen in lines:
        q.put(sen.strip())
    readThread = Thread(target=readQueue, args=(proc, q))
    readThread.daemon = True
    readThread.start()
    print 'Main thread is waiting...\n'
    q.join()
    global notEmpty; notEmpty = False
    print 'Done!'

Upvotes: 1

Views: 3369

Answers (1)

J. P. Petersen
J. P. Petersen

Reputation: 5031

The pipes from the subprocess are file like objects.

The method read() on these objects reads everything into memory until EOF is reached if you don't specify how much is should read, see doc: https://docs.python.org/2/library/stdtypes.html#file.read.

You have to set the size you want to read if you don't want this behaviour. Try setting it to 1 byte?

The same goes for readlines(), see documentation: https://docs.python.org/2/library/stdtypes.html#file.readlines

Upvotes: 1

Related Questions