Unable to pkill a subprocess

Question

Consider:

import subprocess
import time

def run_instance():
    command = "sleep 30 >out 2>err &"
    subprocess.check_output(command, shell=True, start_new_session=True)

def kill_instance():
    print(subprocess.check_output(f"pgrep -f 30", shell=True))
    subprocess.check_output(f"pkill -f 30", shell=True)

run_instance()
time.sleep(1)
kill_instance()

The output (run under Ubuntu) is:

b'17483\n17484\n'
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    kill_instance()
  File "test.py", line 10, in kill_instance
    subprocess.check_output(f"pkill -f 30", shell=True)
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'pkill -f 30' died with <Signals.SIGTERM: 15>.

Why does pkill fail even though pgrep finds the process?

Answer 1

You are vastly overcomplicating things by basically running shell scripts on top of Python.

Here's (something like) the proper solution.

def run_instance():
    command = ["sleep", "30"]
    return subprocess.Popen(command,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL)

def kill_instance(proc):
    proc.kill()

proc = run_instance()
time.sleep(1)
kill_instance(proc)

A more complete example would proc.wait() if the kill failed for whatever reason.

Answer 2

Common way

The most clean, IMO, way here is to just manage your instance within python (note you don't even need shell=True):

import subprocess
import time


def start_instance():
    with open('err', 'w') as err, open('out', 'w') as out:
        return subprocess.Popen(['/usr/bin/sleep', '30'], stdout=out, stderr=err)

def kill_instance(proc):
    proc.kill()

process = start_instance()
time.sleep(1)
kill_instance(process)

Your code

Disclaimer: you shouldn't be doing this in production.

You are killing your pkill. Here's the same with more verbose output:

import subprocess
import time

def run_instance():
    command = "sleep 30 >out 2>err &"
    subprocess.check_output(command, shell=True, start_new_session=True)

def kill_instance():
    print(subprocess.check_output(f"pgrep -f 30 -la", shell=True, universal_newlines=True))  # More verbose output
    subprocess.check_output(f"pkill -f 30", shell=True)

run_instance()
time.sleep(1)
kill_instance()

This outputs on my machine:

2446344 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5434 -container-ip 172.30.0.2 -container-port 5434
2446350 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5434 -container-ip 172.30.0.2 -container-port 5434
2446618 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.30.0.4 -container-port 8080
2446624 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 8080 -container-ip 172.30.0.4 -container-port 8080
2446657 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5001 -container-ip 172.30.0.5 -container-port 5000
2446666 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5001 -container-ip 172.30.0.5 -container-port 5000
2470568 kworker/u8:30-i915
2484058 /usr/lib/firefox/firefox -contentproc -childID 2176 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484060 /usr/lib/firefox/firefox -contentproc -childID 2177 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484067 /usr/lib/firefox/firefox -contentproc -childID 2178 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484121 /usr/lib/firefox/firefox -contentproc -childID 2179 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484283 /usr/lib/firefox/firefox -contentproc -childID 2181 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484293 /usr/lib/firefox/firefox -contentproc -childID 2182 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484392 /usr/lib/firefox/firefox -contentproc -childID 2183 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484394 /usr/lib/firefox/firefox -contentproc -childID 2184 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484423 /usr/lib/firefox/firefox -contentproc -childID 2185 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484442 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220623063721 -prefsLen 41465 -prefMapSize 226778 -appDir /usr/lib/firefox/browser 7462 true socket
2484491 /usr/lib/firefox/firefox -contentproc -childID 2186 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484546 /usr/lib/firefox/firefox -contentproc -childID 2187 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2487379 sleep 30
2487381 /bin/sh -c pgrep -f 30 -la

pkill: killing pid 2446344 failed: Operation not permitted
pkill: killing pid 2446350 failed: Operation not permitted
pkill: killing pid 2446618 failed: Operation not permitted
pkill: killing pid 2446624 failed: Operation not permitted
pkill: killing pid 2446657 failed: Operation not permitted
pkill: killing pid 2446666 failed: Operation not permitted
pkill: killing pid 2470568 failed: Operation not permitted
Traceback (most recent call last):
  File "/tmp/a.py", line 14, in <module>
    kill_instance()
  File "/tmp/a.py", line 10, in kill_instance
    subprocess.check_output(f"pkill -f 30", shell=True)
  File "/usr/local/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'pkill -f 30' died with <Signals.SIGTERM: 15>.

(ignore docker and firefox processes here, they're just for completeness here and to demonstrate that you're killing too much processes, ff tabs crash after this)

The interesting output part from pgrep here:

2487379 sleep 30
2487381 /bin/sh -c pgrep -f 30 -la

When you call pkill, it is a process. It will be there instead of pgrep, something like

2487379 sleep 30
2487xxx /bin/sh -c pkill -f 30

... and you pkill all matching processes. So you can kill the running instance successfully, but then kill yourself.

To demonstrate that, the following succeeds:

import subprocess
import time

def run_instance():
    command = "sleep 30 >out 2>err &"
    subprocess.check_output(command, shell=True, start_new_session=True)

def kill_instance():
    print(subprocess.check_output("pgrep -f 'sleep 30' -la", shell=True, universal_newlines=True))
    subprocess.check_output("pgrep -f 'sleep 30' | grep -v $$ | xargs kill -SIGTERM", shell=True, universal_newlines=True)

run_instance()
time.sleep(1)
kill_instance()

Output:

2508440 sleep 30
2508441 /bin/sh -c pgrep -f 'sleep 30' -la

Sorry for that ugly pipe, maybe there is a simpler way, but Linux pgrep cannot exclude self (as opposed to FreeBSD, discussed here). This pipe lists processes, excludes pgrep itself ($$ refers to current parent shell, which is that runs pgrep here) and passes them to kill.