CODEWITHSUNDEEP
pythonparallel-processingdistributed-systemray
Maybe
Maybe

Reputation: 2279

Parallel: Import a python file from sibling folder

I have a directory tree

working_dir\
    main.py
my_agent\
    my_worker.py
my_utility\
    my_utils.py

Code in each file is as follows

""" main.py """

import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from my_agent.my_worker import MyWorker
import ray

ray.init()
workers = [MyWorker.remote(i) for i in range(10)]
ids = [worker.get_id.remote() for worker in workers]
# print(*ids, sep='\n')
print(*ray.get(ids), sep='\n')

""" worker.py """
from my_utility import my_utils
import ray

@ray.remote
class MyWorker():
    def __init__(self, id):
        self.id = id

    def get_id(self):
        return my_utils.f(self.id)

""" my_utils.py """
def f(id):
    return '{}: Everything is fine...'.format(id)

Here's a part of the error message I received

Traceback (most recent call last):

File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/function_manager.py", line 616, in fetch_and_register_actor unpickled_class = pickle.loads(pickled_class)

File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/cloudpickle/cloudpickle.py", line 894, in subimport import(name)

ImportError: No module named 'my_utility'

Traceback (most recent call last):

File "main.py", line 12, in print(*ray.get(ids), sep='\n')

File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/worker.py", line 2377, in get raise value ray.worker.RayTaskError: ray_worker (pid=30025, host=AiMacbook)

Exception: The actor with name MyWorker failed to be imported, and so cannot execute this method

If I remove all statements related to ray, the above code works fine. Therefore, I boldly guess the reason is that ray runs each actor in a new process and sys.path.append only works in the main process. So I add the following code to worker.py

import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

But it still does not work: the same error message shows up. Now I run out of ideas, what should I do?

Upvotes: 6

Views: 5075

Answers (2)

Archit Kulkarni
Archit Kulkarni

Reputation: 185

The new "Runtime Environments" feature, which didn't exist at the time of this post, should help with this issue: https://docs.ray.io/en/latest/handling-dependencies.html#runtime-environments. (See the working_dir and py_modules entries.)

Upvotes: 3

Robert Nishihara
Robert Nishihara

Reputation: 3362

You are correct about what the issue is.

In your example, you modify sys.path in main.py in order to be able to import my_agent.my_worker and my_utility.my_utils.

However, this path change is not propagated to the worker processes, so if you were to run a remote function like

@ray.remote
def f():
    # Print the PYTHONPATH on the worker process.
    import sys
    print(sys.path)

f.remote()

You would see that sys.path on the worker does not include the parent directory that you added.

The reason that modifying sys.path on the worker (e.g., in the MyWorker constructor) doesn't work is that the MyWorker class definition is pickled and shipped to the workers. Then the worker unpickles it, and the process of unpickling the class definition requires my_utils to be imported, and this fails because the actor constructor hasn't had a chance to run yet.

There are a couple possible solutions here.

  1. Run the script with something like

    PYTHONPATH=$(dirname $(pwd)):$PYTHONPATH python main.py

    (from within working_dir/). That should solve the issue because in this case the worker processes are forked from the scheduler process (which is forked from the main Python interpreter when you call ray.init() and so the environment variable will be inherited by the workers (this doesn't happen for sys.path presumably because it is not an environment variable).

  2. It looks like adding the line

    parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
os.environ["PYTHONPATH"] = parent_dir + ":" + os.environ.get("PYTHONPATH", "")

    in main.py (before the ray.init() call) also works for the same reason as above.

  3. Consider adding a setup.py and installing your project as a Python package so that it's automatically on the relevant path.

Upvotes: 10

Related Questions