CODEWITHSUNDEEP
pythonflasksubprocesspython-multithreading
Edwin199
Edwin199

Reputation: 1

Python Flask, Handling Popen poll / wait / communicate without halting multi-threaded Python

The code below is executed on a certain URL (/new...) and assigns variables to the session cookie, which is used to build the display. This example calls a command using subprocess.Popen.

The problem is that the Popen command called below typically takes 3 minutes - and the subprocess.communicate Waits for the output - during which time all other Flask calls (e.g. another user connecting) are halted. I have some commented lines related to other things I've tried without success - one using the threading module and another using subprocess.poll.

from app import app
from flask import render_template, redirect, session
from subprocess import Popen, PIPE
import threading


@app.route('/new/<number>')
def new_session(number):
    get_list(number)
    #t = threading.Thread(target=get_list, args=(number))
    #t.start()
    #t.join()

    return redirect('/')

def get_list(number):
    #1 Call JAR Get String
    command = 'java -jar fetch.jar' + str(number)
    print "Executing " + command
    stream=Popen(command, shell=False, stdout=PIPE)

    #while stream.poll() is None:
    #    print "stream.poll = " + str(stream.poll())
    #    time.sleep(1)
    stdout,stderr = stream.communicate()

    #do some item splits and some processing, left out for brevity
    session['data'] = stdout.split("\r\n")

    return

What's the "better practice" for handling this situation correctly?

For reference, this code is run in Python 2.7.8 on win32, including Flask 0.10.1

Upvotes: 0

Views: 2325

Answers (1)

Raito
Raito

Reputation: 1573

First, you should use a work queue like Celery, RabbitMQ or Redis (here is a helpful hint).

Then, define the get_list function becomes :

@celery.task
def get_list(number):
    command = 'java -jar fetch.jar {}'.format(number)
    print "Executing " + command
    stream = Popen(command, shell=False, stdout=PIPE)
    stdout, stderr = stream.communicate()
    return stdout.split('\r\n')

And in your view, you wait for the result :

@app.route('/new/<number>')
def new_session(number):
    result = get_list.delay(number)
    session['data'] = result.wait()
    return redirect('/')

Now, it doesn't block your view! :)

Upvotes: 3

Related Questions