How to send the output of a long running Python script over a WebSocket?

Question

A user uploads a python file that I need to execute on my server and send back the stdout that's created over a WebSocket. The python file that's executed will run for several minutes and I need to return the stdout over a socket as they are "printed" out in real-time, not at the completion of the script.

I've tried using: Python. Redirect stdout to a socket, but that's not a WebSocket and my React frontend can't connect to it successfully. (if you can solve that, that would also solve my problem)

I've also tried using websocketd but since I can't add sys.stdout.flush() after each of the users' added print statements it doesn't solve my problem.

I've also tried using subprocess's PIPE functionality but that has the same flush issue

async def time(websocket, path):
    while True:
        data = "test"
        await websocket.send(data)
        # Run subprocess to execute python file in here
        # sys.stdout => websocket.send             

start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

This is the python test script I am using:

from time import sleep
for i in range(40):
    print(i)
    sleep(0.1)

Answer 1

this class will serve as a wrapper for the server

import sys


class ServerWrapper:
    def __init__(self, ws):
        self.__ws = ws
        sys.stdout = self

    def write(self, data):
        self.__ws.send(data)

    def close(self):
        sys.stdout = sys.__stdout__

you need to initialize it with your websocket.

the write function is being called every every time print is called (because we changed sys.stdout to our custom output.

then, after you finished executing the script, you can restore the standard output with close

import asyncio
import websockets
import subprocess
import sys


class ServerWrapper:
    def __init__(self, ws):
        self.__ws = ws
        sys.stdout = self

    def write(self, data):
        self.__ws.send(data)

    def close(self):
        sys.stdout = sys.__stdout__


async def time(websocket, path):
    wrapper = ServerWrapper(websocket)
    # get commands and execute them as you would normally do
    # you don't need to worry about reading output and sending it


start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Answer 2

This stand-alone example will

1. Read a python script from a web socket
2. Write the script to the file system
3. Run the script with output buffering disabled
4. Read the script output one line at a time
5. Write each line of output to the web socket

import asyncio
import websockets
import subprocess

async def time(websocket, path):
    script_name = 'script.py'
    script = await websocket.recv()
    with open(script_name, 'w') as script_file:
        script_file.write(script)
    with subprocess.Popen(['python3', '-u', script_name],
                          stdout=subprocess.PIPE,
                          bufsize=1,
                          universal_newlines=True) as process:
        for line in process.stdout:
            line = line.rstrip()
            print(f"line = {line}")
            await websocket.send(line)

start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

I used this javascript code to test the server:

const WebSocket = require('ws');
let socket = new WebSocket("ws://127.0.0.1:5678");

socket.onopen = function(e) {
    let script = '\
import time\n\
for x in range(100):\n\
    print(f"x = {x}")\n\
    time.sleep(0.25)\n\
';
    console.log("sending data...");
    socket.send(script);
    console.log("done.");
};

socket.onmessage = function(event) {
    console.log(event.data.toString());
};

socket.onerror = function(event) {
    console.log(event);
};

The use of Popen is based on an answer to this question:

Read streaming input from subprocess.communicate()

The -u option is passed to python to disable output buffering.