CODEWITHSUNDEEP
pythondask
JRR
JRR

Reputation: 6152

how does dask distribute data to workers from the scheduler?

Is there any documentation about how dask splits and sends data to workers? I wasn't able to find it on the official website.

Upvotes: 3

Views: 182

Answers (1)

MRocklin
MRocklin

Reputation: 57291

If you are interested in data movement policies then this document on data locality may be of interest to you: http://distributed.readthedocs.io/en/latest/locality.html

If you are interested in the message protocol then this blogpost might help: http://matthewrocklin.com/blog/work/2016/04/14/dask-distributed-optimizing-protocol

As a warning, policies and protocols like these are more ephemeral than the programming interface, and so this answer is likely to become stale in time. Still, this should give an idea of the kinds of things that come into consideration.

Upvotes: 0

Related Questions