rsaw
rsaw

Reputation: 134

Dask SSH-Cluster "no module named xxxx" error

I am using Dask SSH-Cluster to setup a cluster. When I try to submit a task on the local machine and receive its response I get the following error:

.....
  File "/home/rohan/Desktop/BA/DistributedSetup/routers/main_router.py", line 36, in start_dask
    x = res.result()
  File "/home/rohan/.pyenv/versions/3.7.3/lib/python3.7/site-packages/distributed/client.py", line 220, in result
    raise exc.with_traceback(tb)
  File "/home/rohan/.pyenv/versions/3.7.3/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 75, in loads
    return pickle.loads(x)
ModuleNotFoundError: No module named 'utils'

It seems that dask does not find the modules (other files) I use in my project. I am assuming that this has something to do with the pythonpath but am not exactly sure how to go about fixing it. utils in the following code is an other file (module) in my project. The code that I use to reproduce the error:

from utils import helper

def run_algo_dask():
    return "abc"


SSHCluster(hosts=['localhost', 'localhost'],
           connect_options=[{'username': 'rohan'}, {'username': 'rohan'}],
           worker_options={
               "nprocs": 1,
               "nthreads": 1
           }, 
           remote_python=["/home/rohan/.pyenv/shims/python3", "/home/rohan/.pyenv/shims/python3"])

client = Client(cluster)
helper()
res = client.submit(run_algo_dask)
x = res.result()

Upvotes: 1

Views: 123

Answers (0)

Related Questions