CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
PaulWebbster
PaulWebbster

Reputation: 1540

How to handle all errors from Python multiple pools?

to demonstrate problem I've prepared simple code:

from multiprocessing import Pool


class MyError(Exception):

    def __str__(self):
        return repr("Error msg: " + self.args[0])

def pool_function(msg):
    print msg
    raise MyError(msg)
    return 0

def some_function():
    my_pool = Pool(2)
    msg = ['first', 'second']

    my_pool.map(pool_function, msg)

if __name__ == '__main__':
    try:
        some_function()
    except MyError, msg:
        print msg

In this particular example as output I get:

first
second
'Error msg: first'

but I need rather:

first
second
'Error msg: first'
'Error msg: second'

Problem is that on the level of the main function I try to get access to all error messages which are unique, rather than handling it on level of pool_function but I get only first error msg. Unfortunately the real code with which I work is much more complex so is rather hard to do something with the structure of this code which is demonstrated in example. I need some clean and straight forward solution to get all error messages and process it on the level of main function.

Thanks in advice for any solutions.

Upvotes: 0

Views: 75

Answers (1)

dragon2fly
dragon2fly

Reputation: 2419

You have to put try~ except in your pool_function not __main__. If no, __main__ will stop after the first except raised and left no chance for the second one to run. This is following what you are trying:

def pool_function(msg):
    print msg
    try:
        raise MyError(msg)
    except:
        return MyError(msg)


def some_function():
    my_pool = Pool(2)
    msg = ['first', 'second']

    return my_pool.map(pool_function, msg)

if __name__ == '__main__':
    try:
       msg= some_function()
    except MyError, msg:
        print msg

It works, but seem not a good way, so:

def pool_function(msg):
    print msg
    try:
        # do something
        raise MyError(msg)
    except:
        return 0,MyError(msg)
    else:
        return 1,# some result

def some_function():
    my_pool = Pool(2)
    msg = ['first', 'second']
    return return my_pool.map(pool_function, msg)

if __name__ == '__main__':
    msg = some_function()
    for result in msg:
        if result[0]:
            # do something when it run successfully
        elif not result[0]:
            print result[1]
            # do something when it got errors

Upvotes: 1

Related Questions