CODEWITHSUNDEEP
pythonsubprocesspopen
Avichal Badaya
Avichal Badaya

Reputation: 3619

poll method of subprocess.popen returning None for long process

I am executing curl command through subprocess. This curl command starts video processing at another server, and waits for the response. Once the process is done, the remote server returns json object. I am checking the status of the subprocess using poll() value, which is None - process not completed, 0- process completed successfully and 1- for error.

I am getting the correct response if the processing takes around 30 mins/or less on remote server, but if the processing is taking more time, I am getting just None value , even though I can see that the remote server has finished processing and already returned the json object.

Can anyone tell me, what could be the possible reason for poll() returning only None after certain time. Thank you in advance.

My Popen object is :

object = subprocess.Popen(str(curlCmd), shell=True,
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE)

and I am calling object.poll() after every 2 seconds to check if the process is successfully completed or not.

Upvotes: 10

Views: 14073

Answers (4)

Tomas
Tomas

Reputation: 1084

To complement hegemon, this code works on 2.7: 

process = subprocess.Popen(cmd, cwd=current_dir, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE )
process._internal_poll(_deadstate='dead')

while timeout > 0:
  if process.poll() is not 'dead':

Upvotes: 1

jfs
jfs

Reputation: 414079

.poll() is None means that the child is still running.

The curl process may hang as soon as it fills its stdout or stderr OS pipe buffer (around 64K on my Linux box) unless you read from your end of the pipe.

That is while you are waiting for the child to finish, the child waits for you to drain the pipe buffer -- deadlock. From the subprocess docs:

This[.wait()] will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

You could use threads, async.io to consume the pipes concurrently.

Upvotes: 8

hegemon
hegemon

Reputation: 6764

In python 2.7 and possibly 3.x can use:

object._internal_poll(_deadstate=127)

instead of regular Popen.poll() as a workaround. This will return 127 instead of None if the process is terminated.

Of course this is an internal module's method and there's no guarantee it will work after Python's library update.

Upvotes: 1

Avichal Badaya
Avichal Badaya

Reputation: 3619

Finally I ended up using PyCurl instead of creating a subprocess and calling curl command through it. It seems to be an raised issue bit subprocess , where .poll method returns none after certain time, reason is still unclear. I would like to inform people who are using subprocess - poll method ( without wait/ communicate) to be aware of it, if you are running long process . Thank you J.F. Sebastian and Pranav for your directions.

Upvotes: 0

Related Questions