CODEWITHSUNDEEP
pythonmultithreadingpython-2.7
Bevilaqua
Bevilaqua

Reputation: 760

Python Queue raising Full even when infinite

Well, I am a Java developer who resolved to create an application using python 2.7

Basically I am using a Queue to receive information from a UDP port and another process is responsible to pull these values previously inserted out this Queue. So far so good, the only problem is, the documentation (https://docs.python.org/2/library/queue.html) says that when an infinite Queue is aimed, basically we should send a parameter <= 0 when instantiating the Queue (e.g Queue.Queue(maxsize=0)), however when my Queue comprises around 32766 values, it throws/raises a Full exception..

Could anyone tell me why, please?

    while True:
        try:
            self.queue.put(valueToInsertIntoQueue, block=False)
            break
        except Full:
            print('WHYYY? Queue is supposed to be infinite, at least until my memory blows...')
            continue

TraceBack:

Traceback (most recent call last):
  File "MyFile.py", line 71, in inserValueIntoQueue
    self.queue.put(valueToInsertIntoQueue, block=False)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 155, in put
    return self.put(obj, False)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 102, in put
    raise Full
Full

Plataform:

import platform

platform.architecture()

('64bit', '')

Upvotes: 0

Views: 1458

Answers (1)

JGut
JGut

Reputation: 532

Hmm, the fact that it's failing at around 2^15 - 1 leads me to believe that it's something wrong with an underlying variable within the queue. After doing a little research, it seems like it might be something wrong with the pipe, although I'm not 100% sure. Here's a couple of links you might find useful:

http://bugs.python.org/issue8426

http://bugs.python.org/issue8237

Sorry for not answering fully -- I would have commented, but I don't have enough reputation

Edit: I took a look at the source code for multiprocessing queues.

The values in a multiprocessing queue in Python are stored in a Python Deque data structure. Taking a look at the underlying C source code for the deque, I noticed this type definition for the deque:

typedef struct {
    PyObject_VAR_HEAD
    block *leftblock;
    block *rightblock;
    Py_ssize_t leftindex;       /* 0 <= leftindex < BLOCKLEN */
    Py_ssize_t rightindex;      /* 0 <= rightindex < BLOCKLEN */
    size_t state;               /* incremented whenever the indices move */
    Py_ssize_t maxlen;
    PyObject *weakreflist;
} dequeobject;

Specifically, notice this line:

Py_ssize_t maxlen;

According to this PEP,

A new type Py_ssize_t is introduced, which has the same size as the compiler's size_t type, but is signed.

So, my guess would be that your compiler's size_t type is 32 bits. A signed 32 bit integer has a range of values between -32,768 to 32,767, the upper bound being where your program is throwing an exception.

Are you by any chance using 32-bit Python? :)

Upvotes: 3

Related Questions