CODEWITHSUNDEEP
pythongeneratoryield
dorvak
dorvak

Reputation: 9709

Non-blocking generator on Python

I'm using a generator function from the requests module in a QT-Application, pretty much the same as in the requests-streaming example:

import json
import requests

def get_stream():
    r = requests.get('http://httpbin.org/stream/20', stream=True)
    for line in r.iter_lines():
        if line:
            yield json.loads(line)

def consume_stream():
    for message in get_stream():
       #do something

However, when there is no incoming response (f.e. irregularly incoming tweets from Twitters Streaming API), the generator get_stream will block the consume_stream method.

This might occur in any situation where a generator does not yield immediately, but hast to wait for incoming messages etc., and therefore blocks the consumer.

Is there any pattern in Python where you can consume a generator in a non-blocking way, i.e. if the generator yields, process it's results, otherwise do something else until the next results are occuring?

Upvotes: 4

Views: 5752

Answers (4)

user773093
user773093

Reputation: 149

You could use async generators as of python 3.6 https://www.python.org/dev/peps/pep-0525/

import json
import requests

async def get_stream():
    r = requests.get('http://httpbin.org/stream/20', stream=True)
    for line in r.iter_lines():
        if line:
            yield json.loads(line)

async def consume_stream():
   await for message in get_stream():
       #do something

Upvotes: 0

Nathan Davis
Nathan Davis

Reputation: 5766

If you control the generator function, one solution would be to have it throw an exception after a timeout period. Perhaps something along the lines of:

def get_stream(timeout=None):
    while message=read_message(timeout=timout):
        yield message

Then have read_message throw a TimeOutException or something if a timeout condition occurs.

Of course, you still have to deal with the logistics of when/how to retry/resume.

Upvotes: 1

Achim
Achim

Reputation: 15702

As Simeon is asking in the comment, it cannot work as simple as you describe it in your example. There are quite some details you have to take care of. There are different solutions, which make more or less sense, depending on your use case. You don't give much details about what you really want to do, so I will just send you to http://twistedmatrix.com/trac/wiki/QTReactor as an example. There a different solutions/frameworks which implement async message queues. And I think, that's what you are looking for.

Upvotes: 1

shx2
shx2

Reputation: 64318

Take a look at the producer-consumer pattern. It is commonly implemented in python using a Queue.

The producer, typically running in a thread or another process (Queue supports either), simply puts messages in the queue. The consumer, whenever it feels like it, pops messages from the queue. This operation supports a timeout argument.

Upvotes: 2

Related Questions