CODEWITHSUNDEEP
python-3.xparallel-processingcluster-computingdaskdask-distributed
Michael Hansen
Michael Hansen

Reputation: 247

dask worker cannot import module

I am running a dask cluster and a worker w. 16 cores using the CLI utilities. In general it seems to work very well. However, for some reason it will not import modules in the cwd. I try to run the following from my notebook instance:

def tstimp():
  import os
  return os.listdir()
c.run(tstimp)

And i get the following output:

{'tcp://192.168.1.90:35885': ['class_positions.csv',
'.gitignore',
'README.md',
'fullrun.ipynb',
'.git',
'rf.py',
'__pycache__',
'dask-worker-space',
'utils.py',
'.ipynb_checkpoints']}

Note that the module rf.py is listed here. Thus it should be possible to import it in the worker, but when i run the following code:

def tstimp():
  import rf
  return 42
c.run(tstimp)

I get this error: ModuleNotFoundError: No module named 'rf'

Why am I getting this error?

Upvotes: 2

Views: 1554

Answers (1)

Ivo Merchiers
Ivo Merchiers

Reputation: 1688

It seems like the current directory is not added to the python path of the workers. You should be able to fix this by adding it to the path.

def tstimp():
  import sys
  sys.path.append('.')
  import rf
  return 42
c.run(tstimp)

Upvotes: 3

Related Questions