CODEWITHSUNDEEP
parallel-processingipython
Henry Thornton
Henry Thornton

Reputation: 4577

can ipython parallel operate across different sets of nodes

With N=P+Q compute nodes, can ipython send a message to the distinct R nodes and Q nodes at the same time and for each set of nodes to do some computation in parallel and for both set of nodes to return the result to the controller?

Upvotes: 0

Views: 51

Answers (1)

Matt
Matt

Reputation: 27843

Yes, assuming direct view

from IPython.parallel import Client
c = Client()
c.ids
c[:].apply_sync(lambda : "Hello, World")

result in 

['Hello, World',
 'Hello, World',
 'Hello, World',
 'Hello, World',
 'Hello, World',
 'Hello, World',
 'Hello, World',
 'Hello, World']

Partitionning on even/odd engines 

(
c[0::2].apply_sync(lambda : "Hello"),
c[1::2].apply_sync(lambda : "World")
)

result in 

(['Hello', 'Hello', 'Hello', 'Hello'], ['World', 'World', 'World', 'World'])

here I apply sync, so it's not exactly at the same time, but if you apply async, you can collect at once after.

You can also push a variable on each set of node that will "decide" which computation to do.

Upvotes: 1

Related Questions