CODEWITHSUNDEEP
pythonmultithreadingselenium
Mr. Bing
Mr. Bing

Reputation: 235

Unable to kill Python subprocess using process.kill() or process.terminate() or os.kill() or using psutil

Using python, I am starting two subprocesses in parallel. One is an HTTP Server, while other is an execution of another program(CustomSimpleHTTPServer.py, which is a python script generated by selenium IDE plugin to open firefox, navigate to a website and do some interactions). On the other hand, I want to stop execution of the first subprocess(the HTTP Server) when the second subprocess is finished executing.

The logic of my code is that the selenium script will open a website. The website will automatically make a few GET calls to my HTTP Server. After the selenium script is finished executing, the HTTP Server is supposed to be closed so that it can log all the captured requests in a file.

Here is my main Python code:

class Myclass(object):

HTTPSERVERPROCESS = ""

    def startHTTPServer(self):
        print "********HTTP Server started*********"
        try:
            self.HTTPSERVERPROCESS=subprocess.Popen('python CustomSimpleHTTPServer.py', \
                            shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
        except Exception as e:
            print "Exception captured while starting HTTP Server process: %s\n" % e

    def startNavigatingFromBrowser(self):
        print "********Opening firefox to start navigation*********"
        try:
            process=subprocess.Popen('python navigationScript.py', \
                            shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
            process.communicate()
            process.wait()
        except Exception as e:
            print "Exception captured starting Browser Navigation process : %s\n" % e
        try:
            if process.returncode==0:
                print "HTTPSERVEPROCESS value: %s" % self.HTTPSERVERPROCESS.returncode
                print self.HTTPSERVERPROCESS
                #self.HTTPSERVERPROCESS.kill()
                #self.HTTPSERVERPROCESS.terminate()
                #self.kill(self.HTTPSERVERPROCESS.pid)
        except Exception as e:
            print "Exception captured while killing HTTP Server process : %s\n" % e

    def kill(self,proc_pid):
        process = psutil.Process(proc_pid)
        for proc in process.get_children(recursive=True):
            proc.kill()
        process.kill()

    def startCapture(self):
        print "********Starting Parallel execution of Server initiation and firefox navigation script*********"
        t1 = threading.Thread(target=self.startHTTPServer())
        t2 = threading.Thread(target=self.startNavigatingFromBrowser())
        t1.start()
        t2.start()
        t2.join()

Note: Execution starts by calling startCapture()

Here is the code for CustomSimpleHTTPServer.py, which is supposed to write the captured requests to logfile.txt upon termination:

import SimpleHTTPServer
import SocketServer

PORT = 5555

class MyHTTPHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    log_file = open('logfile.txt', 'w')
    def log_message(self, format, *args):
        self.log_file.write("%s - - [%s] %s\n" %
                        (self.client_address[0],
                         self.log_date_time_string(),
                         format%args))

Handler = MyHTTPHandler
httpd = SocketServer.TCPServer(("", PORT), Handler)
httpd.serve_forever()

When I use self.HTTPSERVERPROCESS.kill() or self.HTTPSERVERPROCESS.terminate() or os.kill(), I get following in my terminal upon running the main Python code

********Starting Parallel execution of Server initiation and firefox navigation script*********
********SimpleHTTPServer started*********
********Opening firefox to start navigation*********
HTTPSERVEPROCESS value: <subprocess.Popen object at 0x1080f8410>
2459
Exception captured while killing HTTP Server process : [Errno 3] No such process


Process finished with exit code 0

And when I use self.kill(self.HTTPSERVERPROCESS.pid), I get following in my terminal upon running the main Python code 

********Starting Parallel execution of Server initiation and firefox navigation script*********
********SimpleHTTPServer started*********
********Opening firefox to start navigation*********
HTTPSERVEPROCESS value: <subprocess.Popen object at 0x1080f8410>
2459
Exception captured while killing HTTP Server process : 'Process' object has no attribute 'get_children'


Process finished with exit code 0

Neither of the following 3 are able to kill the HTTPServer process:

self.HTTPSERVERPROCESS.kill()
self.HTTPSERVERPROCESS.terminate()
self.kill(self.HTTPSERVERPROCESS.pid)

I know that CustomSimpleHTTPServer.py is correct because when I run it seperately and manually browse to the website, and then manually terminate the CustomSimpleHTTPServer.py script by hitting CTRL-c in terminal, the logs are populated in logfle.txt.

What changes do I make to my code so that it works properly and logs are populated?

Upvotes: 3

Views: 6658

Answers (1)

ErikR
ErikR

Reputation: 52039

You should just use os.kill() to signal processes:

import os
import signal
...
os.kill(the_pid, signal.SIGTERM)   # usually kills processes
os.kill(the_pid, signal.SIGKILL)   # should always kill a process

Also, if you kill the parent process it also usually kills the children.

Update:

I made two small changes to the Server program:

  • Add a call to self.log_file.flush() to make sure log entries are flushed out to the log file.
  • Override allow_reuse_address so that you can reuse the same address shortly after terminating the server. (See this SO question)

File Server:

#!/usr/bin/env python

import SimpleHTTPServer
import SocketServer
PORT = 5555

class MyServer(SocketServer.TCPServer):
    allow_reuse_address = True

class MyHTTPHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    log_file = open('logfile.txt', 'w')
    def log_message(self, format, *args):
        self.log_file.write("%s - - [%s] %s\n" %
                        (self.client_address[0],
                         self.log_date_time_string(),
                         format%args))
        self.log_file.flush()

Handler = MyHTTPHandler
httpd = MyServer(("", PORT), Handler)
httpd.serve_forever()

Here is an simple navigation program (file Navigate):

#!/usr/bin/env python

import requests
import time

URL = "http://localhost:5555/"

def main():
  for x in range(5):
    print "x =", x
    r = requests.get(URL + "asd")
    time.sleep(1)
  print "Quitting"

main()

And here is the main program:

#!/usr/bin/env python

import os
import subprocess
import signal

def main():
  # start web server
  web = subprocess.Popen(["./Server"])
  print "web server pid:", web.pid

  # start navigator
  nav = subprocess.Popen(["./Navigate"])
  print "nav pid: ", nav.pid

  # wait for nav to exit
  nav.wait()
  print "done waiting for nav"
  print "killing web server"
  os.kill(web.pid, signal.SIGTERM )
  web.wait()
  print "server terminated"

main()

Upvotes: 5

Related Questions