Reputation: 134
I am using Dask SSH-Cluster to setup a cluster. When I try to submit a task on the local machine and receive its response I get the following error:
.....
File "/home/rohan/Desktop/BA/DistributedSetup/routers/main_router.py", line 36, in start_dask
x = res.result()
File "/home/rohan/.pyenv/versions/3.7.3/lib/python3.7/site-packages/distributed/client.py", line 220, in result
raise exc.with_traceback(tb)
File "/home/rohan/.pyenv/versions/3.7.3/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 75, in loads
return pickle.loads(x)
ModuleNotFoundError: No module named 'utils'
It seems that dask does not find the modules (other files) I use in my project. I am assuming that this has something to do with the pythonpath but am not exactly sure how to go about fixing it. utils
in the following code is an other file (module) in my project. The code that I use to reproduce the error:
from utils import helper
def run_algo_dask():
return "abc"
SSHCluster(hosts=['localhost', 'localhost'],
connect_options=[{'username': 'rohan'}, {'username': 'rohan'}],
worker_options={
"nprocs": 1,
"nthreads": 1
},
remote_python=["/home/rohan/.pyenv/shims/python3", "/home/rohan/.pyenv/shims/python3"])
client = Client(cluster)
helper()
res = client.submit(run_algo_dask)
x = res.result()
Upvotes: 1
Views: 123