CODEWITHSUNDEEP
pythonprocessmultiprocess
user1686675
user1686675

Reputation: 21

Does the Process and Pool classes share the same workers?

I have a python script that is doing some map reduce-ish ETL. I am not originator of the code but working to analyze/diagnose its runtime for some improvements.

In the package, it uses a "Process":

worker = Process(target=grab_worker)

worker.start()

That does a perpectual FTP loop to extract new files from our CDN, can't include the FTP code but shouldn't be relevant for the question

Later in the code, we create an instance of a worker Pool which runs some asynch functions:

workerpool =  multiprocessing.Pool(processes=4)
# ...
resultobjs[k] = workerpool.apply_async(func, args=fargs)

Again, the underlying code therein should be irrelevant to question I think so not including code yet.

My question is, in Python, once I create the worker Pool will the workers there be "shared" with the Process?

In other words, if I first create 1 worker with process doing something, later in execution when I create workers with a pool class, when the loop returns and tries to run the function registered with the process, will it then use the previously created workers?

Or, instead, does it keep the "hot side hot and the cold side cold" by allowing each class instance to reference only the workers which it has spawned ( Process reuses its one worker previously created, and Pool continues to use its designated workers vs. Process using the workers generated by Pool ).

Upvotes: 2

Views: 180

Answers (1)

unutbu
unutbu

Reputation: 880697

The mp.Process knows nothing about the mp.Pool. So the call to mp.Process will not somehow use a process spawned by mp.Pool.

Upvotes: 1

Related Questions