CODEWITHSUNDEEP
python-3.xmultiprocessingfuturedask
Dror Hilman
Dror Hilman

Reputation: 7457

dask future not updating according to progress

problem...

  1. I am submitting function to dask client and recording the futures keys.

  2. I am using these key to instantiate the futures in a different fucntion.

  3. These futures are stacked in "pending" mode

backbone of the program I am trying:

from dask.distributed import Client, Future
client = Client()

def func1(client, data):
    future = client.submit(some_long_function, data)
    key = future.key
    return key

key = func1(client, data)

def func2(key):
    future = Future(key, client)
    print(future)  # !! Always show "pending" even when the process finished !

How can I use the "key" of a future to get the status of the run and retrieve the results when done ?

(NOTE: I don't want to use the "Future" object itself, since I would like to send this "key" to a client javascript application)

Upvotes: 3

Views: 203

Answers (1)

MRocklin
MRocklin

Reputation: 57281

When the future gets garbage collected in func1 the client sends a message to the scheduler that it no longer needs the result and so it probably never runs. You probably want to call fire_and_forget on the future to ensure that it runs, even if you don't hold a reference to the future.

See these docs:

Upvotes: 1

Related Questions