CODEWITHSUNDEEP
pythonpython-2.7multiprocessinggoogle-compute-engine
tidus95
tidus95

Reputation: 359

python multiprocessing google compute engine

I have a python script which I run on my old quad-core laptop through multiprocessing. The simulation cannot be parallelised, I just run different instances of it on the different cores.

I was considering renting some more powerful cpus on google compute engine. Will I be able to use the same python multiprocessing script just with more cores available?

the script just calls a pool and then apply_async many times

Upvotes: 2

Views: 2417

Answers (2)

Marcin
Marcin

Reputation: 4289

Multiprocessing works the same way on Google's VMs as on your local box (roughly...). In order to use all cores available on a box, do not pass processes argument to multiprocesing.Pool, like:

pool = multiprocessing.Pool(processes=None)

This will make Python create a Pool with multiprocessing.cpu_count() number of processes, cpu_count usually being a number of cores.

Upvotes: 1

dany L
dany L

Reputation: 2654

Yes it should be possible. If you need access to multiple cores on the same instance, as an example the App Engine flexible runtime, which uses Compute Engine as the underlying VM, allows you to configure the number of cores available

Upvotes: 0

Related Questions