CODEWITHSUNDEEP
python-3.xpandasdask
Draconar
Draconar

Reputation: 1195

Terms on the DASK dashboard

By navigating to the dask dashboard, I see these 3 items being displayed

enter image description here

What does 'released', 'memory' and 'processing' exactly mean in this context? When they change from the left to the right side, what is happening with my computations? Released means that the memory was released because the computation is done?

Upvotes: 1

Views: 72

Answers (1)

SultanOrazbayev
SultanOrazbayev

Reputation: 16561

Dask tracks the status/state of every item in the task graph (DAG) and the possible states are:

  • processing: dask received a task and it is being processed by one of the workers, so this refers to active computation happening on a worker

  • waiting: task was received, but it depends on another task that is still being computed, there is no active computation happening for this task

  • memory: worker finished the computation and the result is held on the worker, typically this is a transitory state (so the memory will be cleared as soon as the object is not needed), but it is possible to maintain objects in memory indefinitely with client.persist

  • released: worker released the object from memory, this is usually because all the tasks that need this object received a copy of the object, so there is no longer need to maintain this object in memory (but it is also possible to trigger this manually with future.release())

  • erred: worker encountered an error when computing the task, this is a catch-all for syntax errors, memory errors, IO errors, etc. The details will be available in the worker log

There are two additional values: no-worker state and forgotten status, you can find more details in the official documentation.

Upvotes: 1

Related Questions