CODEWITHSUNDEEP
pythondistributeddask
Bertrand
Bertrand

Reputation: 1064

Dask shared futures with Channels and memory usage

I'm playing with the dask functionality described here "Shared Futures With Channels".

As explained in the document i managed to submit futures and add them to the channel. Then from another client I manage to run through the future and to get the result from them.

In my use case, it's possible to have a lot of futures loaded to the channel. When i look at the scheduler i see that the memory usage growth even and stay high even after that i get the result.

Is there a way to unload the futures from a channel ?

Upvotes: 1

Views: 96

Answers (1)

MRocklin
MRocklin

Reputation: 57281

As of January 2017 there is no mechanism to clear out specific futures from a channel, though this could possibly be added. You might want to raise an issue here.

You might also consider creating a channel with a maximum length using the maxlen= parameter

chan = Channel('my-channel', maxlen=100)

You will need to do this the first time you create the channel.

Upvotes: 1

Related Questions