Building an HTTP API for continuously running python process

Question

TL;DR: I have a beautifully crafted, continuously running piece of Python code controlling and reading out a physics experiment. Now I want to add an HTTP API.

I have written a module which controls the hardware using USB. I can script several types of autonomously operating experiments, but I'd like to control my running experiment over the internet. I like the idea of an HTTP API, and have implemented a proof-of-concept using Flask's development server.

The experiment runs as a single process claiming the USB connection and periodically (every 16 ms) all data is read out. This process can write hardware settings and commands, and reads data and command responses.

I have a few problems choosing the 'correct' way to communicate with this process. It works if the HTTP server only has a single worker. Then, I can use python's multiprocessing.Pipe for communication. Using more-or-less low-level sockets (or things like zeromq) should work, even for request/response, but I have to implement some sort of protocol: send {'cmd': 'set_voltage', 'value': 900} instead of calling hardware.set_voltage(800) (which I can use in the stand-alone scripts). I can use some sort of RPC, but as far as I know they all (SimpleXMLRPCServer, Pyro) use some sort of event loop for the 'server', in this case the process running the experiment, to process requests. But I can't have an event loop waiting for incoming requests; it should be reading out my hardware! I googled around quite a bit, but however I try to rephrase my question, I end up with Celery as the answer, which mostly fires off one job after another, but isn't really about communicating with a long-running process.

I'm confused. I can get this to work, but I fear I'll be reinventing a few wheels. I just want to launch my app in the terminal, open a web browser from anywhere, and monitor and control my experiment.

Update: The following code is a basic example of using the module:

from pysparc.muonlab.muonlab_ii import MuonlabII

muonlab = MuonlabII()
muonlab.select_lifetime_measurement()
muonlab.set_pmt1_voltage(900)
muonlab.set_pmt1_threshold(500)

lifetimes = []
while True:
    data = muonlab.read_lifetime_data()
    if data:
        print "Muon decays detected with lifetimes", data
        lifetimes.extend(data)

The module lives at https://github.com/HiSPARC/pysparc/tree/master/pysparc/muonlab. My current implementation of the HTTP API lives at https://github.com/HiSPARC/pysparc/blob/master/bin/muonlab_with_http_api.

I'm pretty happy with the module (with lots of tests) but the HTTP API runs using Flask's single-threaded development server (which the documentation and the internet tells me is a bad idea) and passes dictionaries through a Pipe as some sort of IPC. I'd love to be able to do something like this in the above script:

while True:
    data = muonlab.read_lifetime_data()
    if data:
        print "Muon decays detected with lifetimes", data
        lifetimes.extend(data)
    process_remote_requests()

where process_remote_requests is a fairly short function to call the muonlab instance or return data. Then, in my Flask views, I'd have something like:

muonlab = RemoteMuonlab()

@app.route('/pmt1_voltage', methods=['GET', 'PUT'])
def get_data():
    if request.method == 'PUT':
        voltage = request.form['voltage']
        muonlab.set_pmt1_voltage(voltage)
    else:
        voltage = muonlab.get_pmt1_voltage()
        return jsonify(voltage=voltage)

Getting the measurement data from the app is perhaps less of a problem, since I could store that in SQLite or something else that handles concurrent access.

Answer 1

But... you do have an IO loop; it runs every 16ms.

You can use BaseHTTPServer.HTTPServer in such a case; just set the timeout attribute to something small. bascially...

class XmlRPCApi:
    def do_something(self):
        print "doing something"

server = SimpleXMLRPCServer(("localhost", 8000))
server.register_instance(XMLRpcAPI())
server.timeout = 0

while True:
    sleep(0.016)
    do_normal_thing()
    x.handle_request()

Edit: python has a built in server, also built on BaseHTTPServer, capable of serving a flask app. since flask.Flask() happens to be a wsgi compliant application, your process_remote_requests() should look like this:

import wsgiref.simple_server
remote_server = wsgire.simple_server('localhost', 8000, app)
# app here is just your Flask() application!

# as before, set timeout to zero so that you can go right back 
# to your event loop if there are no requests to handle
remote_server.timeout = 0

def process_remote_requests():
    remote_server.handle_request()

This works well enough if you have only short running requests; but if you need to handle requests that may possibly take longer than your event loop's normal polling interval, or if you need to handle more requests than you have polls per unit of time, then you can't use this approach, exactly.

You don't necessarily need to fork off another process, though, You can potentially get by using a pool of workers in another thread. roughly:

import threading
import wsgiref.simple_server
remote_server = wsgire.simple_server('localhost', 8000, app)
POOL_SIZE = 10 # or some other value.
pool = [threading.Thread(target=remote_server.serve_forever) for dummy in xrange(POOL_SIZE)]
for thread in pool:
    thread.daemon = True
    thread.start()

while True:
    pass # normal experiment processing here; don't handle requests in this thread.

However; this approach has one major shortcoming, you now have to deal with concurrency! It's not safe to manipulate your program state as freely as you could with the above loop, since you might be, concurrently manipulating that same state in the main thread (or another http server thread). It's up to you to know when this is valid, wrapping each resource with some sort of mutex lock or whatever is appropriate.