CODEWITHSUNDEEP
pythonbashshell
Radu Rădeanu
Radu Rădeanu

Reputation: 2732

Subprocess module from python fails to run bash commands like "<cmd1> | <cmd2>"

The following python script:

#!/usr/bin/env python

import os
cmd = "echo Hello world | cut -d' ' -f1"
test=os.system(cmd)
print(test)

it runs ok (the output is Hello). But when I use subprocess module this one:

#!/usr/bin/env python

import subprocess
cmd = "echo Hello world | cut -d' ' -f1"
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
test = process.communicate()[0]
print (test)

is not ok. The output is Hello world | cut -d' ' -f1 and I expect to be only Hello. How can I correct it?

I saw that in general subprocess module will fail when I'm using a bash command like:

<cmd1> | <cmd2>

Upvotes: 1

Views: 2760

Answers (1)

abarnert
abarnert

Reputation: 366003

This:

echo Hello world | cut -d' ' -f1

… is not actually a command, it's a fragment of shell script. So you need to have the shell execute it.

You can do this just by adding shell=True to the Popen constructor.

The documentation explains how this works. It also explains better ways to do the same thing without the shell. For example:

p1 = Popen(['echo', 'Hello', 'world'], stdout=PIPE)
p2 = Popen(['cut', "-d' '", '-f1'], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()
test = p2.communicate()[0]

Meanwhile, you almost never want to use split on a command line—and in fact, your example shows exactly why you don't want to:

>>> cmd = "echo Hello world | cut -d' ' -f1"
>>> cmd.split()
['echo', 'Hello', 'world', '|', 'cut', "-d'", "'", '-f1']

Notice that it split -d' ' into two arguments, -d' and '.

If you're using shell=True, don't try to split the arguments at all; just pass a string as your cmd:

process = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

If you're not using the shell, the right way to do this is with the shlex module:

>>> shlex.split(cmd)
['echo', 'Hello', 'world', '|', 'cut', '-d ', '-f1']

Notice that the "-d' '" turned into "-d " this time. That may seem odd at first glance, but it's in fact exactly what the shell would do, and what you want; the cut program will get a space as its d option. (In other words, the quotes are for the shell, not for the program the shell runs.)

(The shlex module also has a handle quote function that you can use for the exact opposite purpose: building a command line from a list of arguments for shell=True.)

However, it's usually better to just create an list of arguments in the first place, instead of trying to figure out how to create a string that, when run through shlex.split(), will give you the list you wanted.

Upvotes: 6

Related Questions