CODEWITHSUNDEEP
pythonpopensubprocess
Greg
Greg

Reputation: 47094

Python - How to call bash commands with pipe?

I can run this normally on the command line in Linux:

$ tar c my_dir | md5sum

But when I try to call it with Python I get an error:

>>> subprocess.Popen(['tar','-c','my_dir','|','md5sum'],shell=True)
<subprocess.Popen object at 0x26c0550>
>>> tar: You must specify one of the `-Acdtrux' or `--test-label'  options
Try `tar --help' or `tar --usage' for more information.

Upvotes: 25

Views: 28861

Answers (5)

secretAgent
secretAgent

Reputation: 69

i would try your on python v3.8.10 :

import subprocess
proc1 = subprocess.run(['tar c my_dir'], stdout=subprocess.PIPE, shell=True)
proc2 = subprocess.run(['md5sum'], input=proc1.stdout, stdout=subprocess.PIPE, shell=True)
print(proc2.stdout.decode())

key points (like outline in my solution on related https://stackoverflow.com/a/68323133/12361522):

  • subprocess.run()
  • no splits of bash command and parameters, i.e. ['tar c my_dir']or ["tar c my_dir"]
  • stdout=subprocess.PIPE for all processes
  • input=proc1.stdout chain of output of previous one into input of the next one
  • enable shell shell=True

Upvotes: 3

Dag
Dag

Reputation: 692

What you actually want is to run a shell subprocess with the shell command as a parameter:

>>> subprocess.Popen(['sh', '-c', 'echo hi | md5sum'], stdout=subprocess.PIPE).communicate()
('764efa883dda1e11db47671c4a3bbd9e  -\n', None)

Upvotes: 5

tMC
tMC

Reputation: 19315

>>> from subprocess import Popen,PIPE
>>> import hashlib
>>> proc = Popen(['tar','-c','/etc/hosts'], stdout=PIPE)
>>> stdout, stderr = proc.communicate()
>>> hashlib.md5(stdout).hexdigest()
'a13061c76e2c9366282412f455460889'
>>>

Upvotes: 1

Greg
Greg

Reputation: 47094

Ok, I'm not sure why but this seems to work:

subprocess.call("tar c my_dir | md5sum",shell=True)

Anyone know why the original code doesn't work?

Upvotes: 11

mdeous
mdeous

Reputation: 18019

You have to use subprocess.PIPE, also, to split the command, you should use shlex.split() to prevent strange behaviours in some cases:

from subprocess import Popen, PIPE
from shlex import split
p1 = Popen(split("tar -c mydir"), stdout=PIPE)
p2 = Popen(split("md5sum"), stdin=p1.stdout)

But to make an archive and generate its checksum, you should use Python built-in modules tarfile and hashlib instead of calling shell commands.

Upvotes: 25

Related Questions