CODEWITHSUNDEEP
daskdask-delayed
headsling
headsling

Reputation: 623

Dask Delayed caching

What is the current state of the art with regards to integrating caching into dask-delayed graphs? I have large graphs that have paths that would benefit significantly from persistent caching (i.e. disk, and hashed by params) for each separate run of the graph (i.e. across processing).

My expectation would be to have an api that I can integrate with during the graph optimization phase, replacing nodes with constants loaded from disk where appropriate. It looks like cachey should be my entry point to this, but I have not been able to get it to work with pure dask-delayed examples.

Upvotes: 0

Views: 161

Answers (1)

MRocklin
MRocklin

Reputation: 57261

Yes, cachey could probably handle this. The docpage you're probably referring to is here: http://docs.dask.org/en/latest/caching.html

If there is a particular bug then I encourage you to open a bug report.

Upvotes: -1

Related Questions