CODEWITHSUNDEEP
pythondaskdask-distributed
feik
feik

Reputation: 387

Dask Intermediate Results

I have a smallish Dask custom application (~20 nodes in the DAG). I would like to be able to somehow persist all of the intermediate results of the functions for future inspection, as sometimes we want to know why we reached our final answer. Are there any good patterns for this within Dask other than pushing the results to Redis (or the like) just before returning the function?

Upvotes: 1

Views: 504

Answers (1)

MRocklin
MRocklin

Reputation: 57251

You can compute intermediate results along with your final results.

a = dask.delayed(inc)(1)
b = dask.delayed(inc)(2)
c = dask.delayed(add)(a, b)

dask.compute(c)  # only return c, releasing a and b as soon as possible

dask.compute(a, b, c)  # return all three

Upvotes: 1

Related Questions