Why do I keep getting [BrokenPipeError: [Errno 32] Broken pipe] no matter the number of workers in my Pool with multiprocessing lib in python3.8?

Question

I am trying to use a Pool of workers python3's multiprocessing library to accelerate a CPU intensive task. I notice that no matter the number of workers in my pool, I always end up getting the following error :

BrokenPipeError: [Errno 32] Broken pipe

The code that leads to this error is the following:

def save_result(rincL):
    #Save results here

if __name__ == '__main__':
    p = Pool(8)
    allProcesses = []
    for i in range(60): 
        rincL = RINC_L(2, X_train, y_train[:,i].ravel(), X_test, y_test[:,i].ravel(), 6, i)
        allProcesses.append(p.apply_async(rincL.RINC_N, [0]))
    #Moved the two following lines based on suggestions in the comment
    #Still getting the same error.
    #p.close()
    #p.join()
    for process in allProcesses:
        save_result(process.get())
        time.sleep(1)
    p.close()
    p.join()

I also noticed that after a while of this program running. The number of processes jumps from 8 to a very high number. Something like 30 processes are spawned:

Answer 1

The child processes were using too much memory and were being killed by the OS.