CODEWITHSUNDEEP
pythonqueue
Dominic Bou-Samra
Dominic Bou-Samra

Reputation: 15416

Multiprocessing queue batch get up to max N elements

I need to get as many items as I can from a queue (up to N), in a blocking fashion. e.g:

queue.get(16)

Should return up to 16 elements, but block if empty.

Upvotes: 6

Views: 4090

Answers (2)

mrgloom
mrgloom

Reputation: 21632

It waits 3 sec, if queue is empty. It returns last not full batch. 

def get_batch_from_queue(q, batch_size):
    item_list = []

    try:
        for i in range(batch_size):
            item_list.append(q.get(block=True, timeout=3))

        return item_list
    except queue.Empty:
        if(len(item_list)==0):
            return None
        else:
            return item_list

while True:
    batch = get_batch_from_queue(q, batch_size=8)
    if batch == None:
        break
    print('len(batch)', len(batch))

Upvotes: 0

Tim Peters
Tim Peters

Reputation: 70602

There's no such facility built in, so you'll need to code it yourself; for example,

import queue  # in Python 3; Queue in Python 2
...
def getn(q, n):
    result = [q.get()]  # block until at least 1
    try:  # add more until `q` is empty or `n` items obtained
        while len(result) < n:
            result.append(q.get(block=False))
    except queue.Empty:
        pass
    return result

Then do getn(queue, 16) for your conceptual queue.get(16).

Upvotes: 8

Related Questions