CODEWITHSUNDEEP
pythonmessage-queuepython-multiprocessing
Stefan Bossbaly
Stefan Bossbaly

Reputation: 6794

Multiprocessing Queue Negative Max Size

There are some examples of code that specify a negative number for the maxsize of a multiprocessing.Queue like so

import multiprocessing
queue = multiprocessing.Queue(-1)

I could not find anything in the documentation that specifies what a negative value means so I looked at the multiprocessing.Queue source code.

class Queue(object):

    def __init__(self, maxsize=0):
        if maxsize <= 0:
            maxsize = _multiprocessing.SemLock.SEM_VALUE_MAX
        self._maxsize = maxsize
        ...

I can't seem to find the value for _multiprocessing.SemLock.SEM_VALUE_MAX. Does a negative number create the biggest Queue possible supported by the OS? Is there any documentation that clarifies this magic value?

Upvotes: 0

Views: 1123

Answers (2)

miraculixx
miraculixx

Reputation: 10349

I can't seem to find the value for _multiprocessing.SemLock.SEM_VALUE_MAX

Try this:

python -c "import _multiprocessing; import platform; print platform.platform(), _multiprocessing.SemLock.SEM_VALUE_MAX"

Specific values found so far (please edit and update accordingly):

# OSX 10.9.5
Darwin-13.4.0-x86_64-i386-64bit 32767 
# Linux Mint Qiana
Linux-3.13.0-24-generic-x86_64-with-LinuxMint-17-qiana 2147483647
# Fedora
Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-fedora-21-Twenty_One 2147483647
# Ubuntu Trusty
Linux-3.13.0-45-generic-x86_64-with-Ubuntu-14.04-trusty 2147483647
# Debian 8
Linux-2.6.32.46-i686-with-debian-8.0 2147483647

This may be a good case to suggest an improvement to the documentation. If you would be so kind.

Documentation hints

I could not find anything in the documentation that specifies what a negative value means

It is actually platform dependent, use the above command the get the actual value.

While there does not seem to be documentation on maxsize specific to multiprocessing, the standard library's Queue documentation has this to offer:

If maxsize is less than or equal to zero, the queue size is infinite.

and multiprocessing's guide states that

The Queue class is a near clone of Queue.Queue.

Hence I suppose it is fair to assume that your interpretation is correct, i.e. create the biggest Queue possible supported by the OS. Actually, we don't have to revert to assumptions, it's a fact:

The gory details

If you want the gory details of how _multiprocessing.SemLock.SEM_VALUE_MAX is determined at compile time, read on.

Is there any documentation that clarifies this magic value?

No -- If you really want to know, you have to follow the code... For CPython there are these #include files multiprocessing.h and the platform C library's limits.h, the latter eventually being included by Python.h. In other words, SEM_VALUE_MAX is set depending on the host platform.

As per Open Group's documentation on limits.h It is is defined as

The maximum value a semaphore may have. Minimum Acceptable Value: _POSIX_SEM_VALUE_MAX

So _POSIX_SEM_VALUE_MAX is the minimum which in turn is defined as:

The maximum value a semaphore may have. Value: 32 767

Upvotes: 1

abarnert
abarnert

Reputation: 365767

The docs for multiprocessing.Queue apparently don't say this directly, but it's strongly implied in two places.

First, the Queue docs say:

Queue implements all the methods of queue.Queue except for task_done() and join().

And earlier, in Exchanging objects between processes:

The Queue class is a near clone of queue.Queue.

And, if you follow the link:

Constructor for a FIFO queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.

So, yes, -1 is documented to mean infinite… although maybe not in the most direct way possible.

You could file a doc bug on this. Maybe all of the classes should explicitly say "The interface is exactly like <relevant other class, with link> except as specified below", instead of leaving you to find that information elsewhere in the docs?

Since you actually asked two questions, not one, here's where SEM_VALUE_MAX is defined:

In PyInit__multiprocessing, the top-level code for the _multiprocessing module, it's added dynamically to the __dict__ of the SemLock type.

The value it gets set to is platform-specific, but it's generally going to be your platform's SEM_VALUE_MAX or INT_MAX. The former is defined in POSIX 2004 as part of limits.h, and just specifies:

The maximum value a semaphore may have.

The source comments may be a bit confusing.

I don't know about NetBSD, but IIRC, old FreeBSD did the same: the value is provided as a signed integer (actually, it's just a #define preprocessor macro, but that makes it an int-typed constant), but the APIs that use it take unsigned values. In C, you can just pass -1 to a function that takes an unsigned int and it's the same as passing UINT_MAX (4294967295). Python doesn't have those casting rules, so the module just converts a -1 to INT_MAX (2147483647), which is safer, and you're not going to have more than 2.1 billion semaphores. I believe current FreeBSD and its descendants, like OS X, just give you USHORT_MAX (32767), which makes this unnecessary, but there may be other OS's that do the same.

Upvotes: 3

Related Questions