Program works with map() but raises TypeError with pool.map()

Question

I do have a program, which has not been used for quite a while, but it has been used and worked already. It uses multiprocessing as the same task has to be done for different data many times.

I now touched the program to add a new parameter, tested it and notices that it comes up with an error. Also the earlier (version controlled) version comes up with the same error. The full error looks like this:

Exception in thread Thread-2:
    Traceback (most recent call last):
        File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
            self.run()
        File "/usr/lib64/python2.7/threading.py", line 764, in run
            self.__target(*self.__args, **self.__kwargs)
        File "/usr/lib64/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
           put(task)
TypeError: 'NoneType' object is not callable

That is all. it does not say much to me, to be honest. While trying to debug this, i came to the idea to try using the normal map() instead of the pooled version pool.map. The script then runs fine.

I can not come up with a minimal example, which reproduces the error, but I can come up with an example, where everything works fine, just as expected:

import random
import time
from multiprocessing import Pool


def do_work(x, y, z):
     time.sleep(random.random() * 2)
     print x + y + z

def do_one(arguments):
    print "doing one"
    do_work(*arguments)

def do_many(x, y, zs):
    map(do_one, [(x, y, z) for z in zs])

def do_many_pooled(x, y, zs):
    pool = Pool(2)
    pool.map(do_one, [(x, y, z) for z in zs])
    pool.close()
    pool.join()

def main():
    x = 1
    y = 2
    zs = range(10)
    print "doing many"
    do_many(x, y, zs)
    print "doing many pooled"
    do_many_pooled(x, y, zs)


if __name__ == '__main__':
    main()

The real program does a lot of database requests, computations using numpy and storing results back into a database. In the real program, the program exits with an error, before printing "doing one", when used with the pooled version, but runs fine using the non-pooled version.

Does anyone know, how to read the Traceback correctly and/or can tell me, what possibly can cause this Exception?

Answer 1

I'd say it looks like put is being clobbered an set to None from this part of the traceback:

File "/usr/lib64/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
   put(task)
TypeError: 'NoneType' object is not callable

Looking at the Python source, Pool.__init__() is setting up _task_handler which will make a call to _handle_tasks and is providing the arguments for said call:

self._task_handler = threading.Thread(
    target=Pool._handle_tasks,
    args=(self._taskqueue, self._quick_put, self._outqueue, self._pool)
    )

If you look at _handle_tasks, then you see that self._quick_put is what ends up being the put variable:

@staticmethod
def _handle_tasks(taskqueue, put, outqueue, pool, cache):
    thread = threading.current_thread()

    for taskseq, set_length in iter(taskqueue.get, None):
        i = -1
        for i, task in enumerate(taskseq):
            if thread._state:
                debug('task handler found thread._state != RUN')
                break
            try:
                put(task)
            except Exception as e:
                job, ind = task[:2]
                try:
                    cache[job]._set(ind, (False, e))
                except KeyError:
                    pass
        else:
            if set_length:
                debug('doing set_length()')
                set_length(i+1)
            continue
        break
    else:
        debug('task handler got sentinel')

Moreover, you can see that all exceptions are being caught here and tucked away for reporting later. But, if you head back to Python 2.7.6, you'll see this:

@staticmethod
def _handle_tasks(taskqueue, put, outqueue, pool):
    thread = threading.current_thread()

    for taskseq, set_length in iter(taskqueue.get, None):
        i = -1
        for i, task in enumerate(taskseq):
            if thread._state:
                debug('task handler found thread._state != RUN')
                break
            try:
                put(task)
            except IOError:
                debug('could not put task on queue')
                break
        else:
            if set_length:
                debug('doing set_length()')
                set_length(i+1)
            continue
        break
    else:
        debug('task handler got sentinel')

Notice here that a TypeError could escape. It turns out that this was fixed as a result of bug #19425. Oddly, it was claimed that this was not an issue in Python 2.7, but the changeset was still backported.

At any rate, in either case, put() should be a known value, and there doesn't appear to be any way to set put in this code. So, to me, it smells like a bug in Python. Any chance you can run the same code under a newer version of Python?

Some other useful information

A quick Google search returns some interesting results too:

• Python bug #9755 - Similar but different stack trace.

• Python bug #15881 - Similar to the above, but a slightly different stack trace too.

Both are issues where cleanup was affecting module state and causing things to fail with a "'NoneType' object is not callable" exception.

Other, somewhat related bugs have existed in Python too. At one point you could encounter a similar exception with the use of daemon threads and exiting the main thread of your application. I forget which version that was fixed in. I just wanted to show that this kind of problem isn't unheard of and is a bug in Python.