CODEWITHSUNDEEP
pythonparallel-processing
Bill
Bill

Reputation: 15

AttributeError: Can't pickle local object 'parallel_operations.<locals>.process'

def parallel_operations(points, primitives):

    batch_size,number_of_points,_ = points.shape
    _,_,number_of_primitives = primitives.shape
    gradient = torch.zeros(batch_size,number_of_points,number_of_primitives)
    def process(_lock,i):
        
        _lock.acquire()
        temp_points = points[i,:,:]
        
        temp_primitives= primitives[i,:,:].transpose(1,0) #[7,1024]
        #print("temp_shape{}".format(temp_primitives.shape))
        
        temp = torch.zeros(number_of_points,number_of_primitives)
        for k in range(number_of_points):
            for j in range(number_of_primitives):
                temp[k,j] = torch.norm(temp_points[k,:]*temp_primitives[j,:3]+temp_primitives[j,3:6])
        gradient[i,:,:] = temp
        
        
        print("gradient update {} {}".format(i, gradient))
        lock.release()
        return (i, gradient[i,:,:])
    result = []
    pool = Pool(multiprocessing.cpu_count())
    lock = Manager().Lock()
    for i in range(10):
        result.append(pool.apply_async(process,args=(lock,i)))
    pool.close()    
    pool.join()
    print(len(result))
    for i in result:
        print(i.get())if __name__ == "__main__":

    points = torch.randn(10,3,3)
    primitives = torch.randn(10,7,3)
    result1 = parallel_operations(points,primitives)

The above is my parellized code but when I run it, it throw an error :`AttributeError: Can't pickle local object 'parallel_operations.<locals>.process' . Why is that?

Upvotes: 1

Views: 985

Answers (1)

tdelaney
tdelaney

Reputation: 77387

When multiprocessing executes a function in a subprocess, it serializes the function and its parameters via the pickle protocol. But pickle doesn't serialize the code object itself, just its module and name. The unpickler loads the module and can get the right function via its name in that module.

But inner functions can't be reached by name like that. They are compiled once but they are only assigned to a function's local variable namespace when the function is executed. Otherwise, they are anonymous objects known to the function byte code. You can see this by disassembling a very simple program:

from dis import dis

def foo():

    def bar(x):
        return x

print(dis(foo))

The output is

  5           0 LOAD_CONST               1 (<code object bar at 0x7f18c7f5b890, file "/home/td/tmp/l/w1.py", line 5>)
              2 LOAD_CONST               2 ('foo.<locals>.bar')
              4 MAKE_FUNCTION            0
              6 STORE_FAST               0 (bar)
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

Disassembly of <code object bar at 0x7f18c7f5b890, file "/home/td/tmp/l/w1.py", line 5>:
  6           0 LOAD_FAST                0 (x)
              2 RETURN_VALUE

The first section is the outer function. It binds an anonymous code object to local variable "bar" each time the function is run. When no instance of the outer function is running, the inner function has no name. The second section is the anonymous code object itself.

You should move process outside of parallel_operations so that pickle can find it.

Upvotes: 1

Related Questions