CODEWITHSUNDEEP
pythonmultiprocessingpickle
mysterious_guy
mysterious_guy

Reputation: 435

Pickling error while using pool.map in multiprocessing

I have a function which is as follows:

def func1(date1,date2):
 #calculations

if __name__=='__main__':
    pool = Pool(processes=4)
    dates = [[dt.datetime(2016,6,17),dt.datetime(2016,6,23)],[dt.datetime(2016,6,24),dt.datetime(2016,6,30)],[dt.datetime(2016,7,1),dt.datetime(2016,7,7)],[dt.datetime(2016,7,8),dt.datetime(2016,7,14)]]
    result=pool.map(lambda x: func1(x),dates)

I see below error:

    File "Location_Factors_C.py", line 204, in <module>
    result=pool.map(lambda x: func1(x),dates)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Upvotes: 0

Views: 785

Answers (2)

mysterious_guy
mysterious_guy

Reputation: 435

Changed my function to accept a list of dates :

def func1(datelist):
   date1 = datelist[0]
   date2 = datelist[1]


if __name__=='__main__':
    pool = Pool(processes=4)
    dates = [[dt.datetime(2016,6,17),dt.datetime(2016,6,23)],[dt.datetime(2016,6,24),dt.datetime(2016,6,30)],[dt.datetime(2016,7,1),dt.datetime(2016,7,7)],[dt.datetime(2016,7,8),dt.datetime(2016,7,14)]]
    result=pool.map(func1,dates)

Upvotes: 0

Back2Basics
Back2Basics

Reputation: 7806

looks like you need an asterisk

result=pool.map(lambda x: func1(x),dates)

should be:

result=pool.map(lambda x: func1(*x),dates)

You have func1() needing 2 arguments and you are only passing a single list in.

Upvotes: 1

Related Questions