CODEWITHSUNDEEP
pythonpandasdaskdistributeddask-dataframe
dytyniak
dytyniak

Reputation: 394

Is it possible to have default distributed Client per thread?

I have 2 independent remote clusters and running few parallel tasks in different threads that contain Dask DataFrame manipulations. I don't want explicitly use client in each thread - client.compute(df). I want to use default - df.compute().

Is it possible to set default client for each thread?

Upvotes: 2

Views: 52

Answers (1)

SultanOrazbayev
SultanOrazbayev

Reputation: 16561

This might not be the solution you are looking for, but one option is to wrap relevant computations in a context manager:

from distributed import Client, LocalCluster
from dask.datasets import timeseries

cluster1 = LocalCluster(n_workers=1, dashboard_address=':0')
cluster2 = LocalCluster(n_workers=1, dashboard_address=':0')

with Client(cluster1) as client:
    print(timeseries().head())

with Client(cluster2) as client:
    print(timeseries().head())

Upvotes: 2

Related Questions