How to use a Python script to start and stop a Python script containing "http.server.serveforever"

Question

Background

I have a Python 3 script called server.py that uses the built-in http.server module. The script boils down to the following:

from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler

class MyRequestHandler (BaseHTTPRequestHandler):
    def do_POST(self):
        # Code omitted for brevity

    def do_GET(self):
        # Code omitted for brevity

def start_server():

    # Begin serving
    # -------------
    server = HTTPServer(('', port), MyRequestHandler)
    print("server now running on port {0} ...".format(port))

    server.serve_forever()

# Start the Server
# ----------------
if __name__ == '__main__':
    start_server()

MyRequestHandler handles GET and POST requests by importing modules "on-the-fly" depending on the URI that was used for the request.

The above works fine, however, subsequent to the creation of that script, there has been a requirement to be able to remotely update the whole "package" of scripts (i.e. the server script, plus all of the "module scripts" that are loaded up "on-the-fly" and are located in sub-folders).

For this, I have written another server script in Python 3 (called updater.py) which, when instructed, will retrieve a zip file, which it then unzips to overwrite the original server.py script along with all the other associated scripts and sub-folders.

Question

This all works great, but I have now hit a wall. The best approach to this, I thought, would be to have the updater.py script control the running of server.py. It could shut down the server.py, and everything linked to it before overwriting it all and then give it a clean start after it is overwritten.

On that basis, the road that I have gone down is to use subprocess.Popen to start the server, believing that I could just kill the Python process before overwriting the server.py stuff, however, this is not working as hoped. Here is my trial.py script that I have written to test the theory:

import sys
import subprocess

def main():
    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'])
        print("Started process:")
        print(proc.pid)
        return proc

    def kill_process(the_process):
        print("Killing process:")
        print(the_process.pid)
        the_process.kill()

    process = None

    while True:
        user_input = input("Type something: ")

        if user_input == 'start':
            process = start_process()
        if user_input == 'kill':
            kill_process(process)
        if user_input == 'exit':
            break

if __name__ == '__main__':
    main()

This does appear to start and kill a Python process, but the server is not running while this script is running, so I am not sure what it is starting and killing! Typing "start" and then "exit" (and thus quitting the trial.py script) allows the server to run, though I can't understand why, since I thought subprocess.Popen should cause the spawned process to run independently of the parent process?

edit: Thanks to @Håken Lid's astute observation below, I noticed that all I was doing is breaking out of the while loop, not exiting the script. This leads me to believe that the while loop is somehow blocking the sub-process from running (since once the loop is exited, the server will start).

Answer 1

per our discussion, I'd recommend some way to empty the stdio buffers from "server.py". If you want to also be able to give user input, you'll need a thread to do the printing (or just to empty the buffers into a black hole) while you wait for user input on the main thread. Here's a rough idea of how I might do it..

import sys
import subprocess
from threading import Thread 
#This could probably be solved with async, but I still
#haven't learned async as well as I know threads

def main():

    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'], 
                                stdin=subprocess.PIPE, 
                                stdout=subprocess.PIPE)
        print("Started process:")

        def buf_readerd(proc, inbuf, outbuf):
            while proc.poll() is None:
                outbuf.write(inbuf.readline()) #may need to add a newline.. I'm not sure if readline ends in a \n

        stdoutd = Thread(target=buf_readerd, args=(proc, proc.stdout, sys.stdout), daemon=True)
        stderrd = Thread(target=buf_readerd, args=(proc, proc.stderr, sys.stderr), daemon=True)
        stdoutd.start()
        stderrd.start()
        print("started pipe reader daemons")

        print(proc.pid)
        return proc
# ...

Answer 2

Okay, so I think I have fixed this myself now. The problem was that I was blocking the sub-process execution with the call to input() inside the while loop. The script does nothing until it receives some input. I made subprocess.poll() an integral part of the loop and put the call to input() inside this loop. If I hit <return> a few times, the server will start and I can use it. If I then type kill, the server will be killed, as expected.

I am not sure how this will work in the context of my updater.py script, but I have learnt a valuable lesson in "thinking more deeply before posting to StackOverflow"!

import sys
import subprocess

def main():
    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'])
        print("Started process:")
        print(proc.pid)
        return proc

    def kill_process(the_process):
        print("Killing process:")
        print(the_process.pid)
        the_process.kill()

    user_input = input("Type something: ")

    if user_input == 'start':
        process = start_process()

        while process.poll() is None:
            user_input = input()
            if user_input == 'kill' or user_input == 'exit':
                kill_process(process)

if __name__ == '__main__':
    main()