CODEWITHSUNDEEP
resourcesdaskdask-delayed
ranjith
ranjith

Reputation: 135

Different resource allocations for each of the task in a list

I have a list of independent tasks and each needs different resources and takes different calculation times. I have to specify resource constraints on each of these tasks in the list and set the priority for the task with the least amount of resources to be finished first.

@dask.delayed()
def func(...)
...
 
task_list = []
...
   task_list.append(dask.delayed(func)(...))

dask.compute(*task_list, resources={'memory':10e9}, optmize_graph=False)

In the above code, I could assign a resource constraint on all the tasks in the list but instead, i would like to assign different resource constraints for each of the tasks in the list.

something like this:

task_list = [delayed(func1..), delayed(func2) .....]

delayed(func1) - {'memory':10e9} # this task needs 10 GB of memory to start
delayed(func2): {'memory':5e9} # # this task needs 5 GB of memory to start
...

dask.compute(*task_list)

Could someone please help.

Thanks !

Upvotes: 0

Views: 193

Answers (1)

MRocklin
MRocklin

Reputation: 57301

As of 2020-08-07 Dask does not support this. It is a frequently requested feature though. You may want to go through the github issues to see if you can find similar people and share your thoughts.

Upvotes: 1

Related Questions