CODEWITHSUNDEEP
pythonbashshellsubprocessquotes
Chris Chambers
Chris Chambers

Reputation: 1365

Python Subprocess: Unable to Escape Quotes

I know similar questions have been asked before, but they all seem to have been resolved by reworking how arguments are passed (i.e. using a list, etc).

However, I have a problem here in that I don't have that option. There is a particular command line program (I am using a Bash shell) to which I must pass a quoted string. It cannot be unquoted, it cannot have a replicated argument, it just has to be either single or double quoted.

command -flag 'foo foo1'

I cannot use command -flag foo foo1, nor can I use command -flag foo -flag foo1. I believe this is an oversight in how the command was programmed to receive input, but I have no control over it.

I am passing arguments as follows:

self.commands = [
                self.path,
                '-flag1', quoted_argument,
                '-flag2', 'test',
                ...etc...
                ]
process = subprocess.Popen(self.commands, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
results = process.communicate(input)

Where quoted_argument is something like 'foo foo1 foo2'. I have tried escaping the single quote ("\'foo foo1 foo2\'"), but I get no output.

I know this is considered bad practice because it is ambiguous to interpret, but I don't have another option. Any ideas?

Upvotes: 2

Views: 7440

Answers (2)

tdelaney
tdelaney

Reputation: 77347

The shell breaks command strings into lists. The quotes tell the shell to put multiple words into a single list item. Since you are building the list yourself, you add the words as a single item without the quotes.

These two Popen commands are equivalent

Popen("command -flag 'foo foo1'", shell=True)
Popen(["command", "-flag", "foo foo1"])

EDIT

This answer deals with escaping characters in the shell. If you don't use the shell, you don't add any quotes or escapes, just put in the string itself. There are other issues with skipping the shell, like piping commands, running background jobs, using shell variables and etc. These all can be done in python instead of the shell.

Upvotes: 8

Flimm
Flimm

Reputation: 150653

A mental model of process and shells that I found very helpful:

This mental model has helped me a lot through the years.

Processes in your operating system receive an array of strings representing the arguments. In Python, this array can be accessed from sys.argv. In C, this is the argv array passed to the main function. And so on.

When you open a terminal, you are running a shell inside that terminal, for example bash or zsh. What happens if you run a command like this one?

$ /usr/bin/touch one two

What happens is that the shell interprets the command that you wrote and splits it by whitespace to create the array ["/usr/bin/touch", "one", "two"]. It then launches a new process using that list of arguments, in this case creating two files named one and two.

What if you wanted one file named one two with a space? You can't pass the shell a list of arguments as you might want to do, you can only pass it a string. Shells like Bash and Zsh use single quotes to workaround this:

$ /usr/bin/touch 'one two'

The shell will create a new process with the arguments ["/usr/bin/touch", "one two"], which in this case create a file named one two.

Shells have special features like piping. With a shell, you can do something like this:

$ /usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z
THIS IS AN EXAMPLE

In this case, the shell interprets the | character differently. In creates a process with the arguments ["/usr/bin/echo", "This is an example"] and another process with the arguments ["/usr/bin/tr", "a-z", "A-Z"], and will pipe the output of the former to the input of the latter.

How this applies to subprocess in Python

Now, in Python, you can use subprocess with shell=False (which is the default, or with shell=True. If you use the default behaviour shell=False, then subprocess expects you to pass it a list of arguments. You cannot use special shell features like shell piping. On the plus side, you don't have to worry about escaping special characters for the shell:

import subprocess
# create a file named "one two"
subprocess.call(["/usr/bin/touch", "one two"])

If you do want to use shell features, you can do something like:

subprocess.call(
    "/usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z",
    shell=True,
)

If you are using variables with no particular guarantees, remember to escape the command:

import shlex
import subprocess

subprocess.call(
    "/usr/bin/echo " + shlex.quote(variable) + " | /usr/bin/tr a-z A-Z",
    shell=True,
)

(Note that shlex.quote is only designed for UNIX shells, and not for DOS on Windows.)

Upvotes: 2

Related Questions