Ray on AWS: Could not find any running Ray instance

Question

• Installed recent Ray (ray==1.12.0)
• Followed precisely AWS cluster instructions from https://docs.ray.io/en/latest/cluster/quickstart.html - ray up succeeds, but gives one exception:

2022-04-28 08:19:46,218 ERROR services.py:1481 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
 The last 10 lines of /tmp/ray/session_2022-04-28_08-19-43_178339_1674/logs/dashboard.log:
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/dashboard/modules/state/state_head.py", line 11, in <module>
    from ray.dashboard.state_aggregator import StateAPIManager
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/dashboard/state_aggregator.py", line 21, in <module>
    from ray.experimental.state.state_manager import StateDataSourceClient
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/experimental/state/state_manager.py", line 67, in <module>
    class StateDataSourceClient:
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/experimental/state/state_manager.py", line 80, in StateDataSourceClient
    def __init__(self, gcs_channel: grpc.aio.Channel):
AttributeError: module 'grpc' has no attribute 'aio'
2022-04-28 08:19:46,218 ERROR services.py:1482 -- Failed to start the dashboard, return code 1
 The last 10 lines of /tmp/ray/session_2022-04-28_08-19-43_178339_1674/logs/dashboard.log:
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/dashboard/modules/state/state_head.py", line 11, in <module>
    from ray.dashboard.state_aggregator import StateAPIManager
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/dashboard/state_aggregator.py", line 21, in <module>
    from ray.experimental.state.state_manager import StateDataSourceClient
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/experimental/state/state_manager.py", line 67, in <module>
    class StateDataSourceClient:
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/experimental/state/state_manager.py", line 80, in StateDataSourceClient
    def __init__(self, gcs_channel: grpc.aio.Channel):
AttributeError: module 'grpc' has no attribute 'aio'
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/_private/services.py", line 1458, in start_dashboard
    raise Exception(err_msg + last_log_str)
Exception: Failed to start the dashboard, return code 1
 The last 10 lines of /tmp/ray/session_2022-04-28_08-19-43_178339_1674/logs/dashboard.log:
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/dashboard/modules/state/state_head.py", line 11, in <module>
    from ray.dashboard.state_aggregator import StateAPIManager
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/dashboard/state_aggregator.py", line 21, in <module>
    from ray.experimental.state.state_manager import StateDataSourceClient
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/experimental/state/state_manager.py", line 67, in <module>
    class StateDataSourceClient:
  File "/home/ubuntu/anaconda3/envs/tensorflow2_latest_p37/lib/python3.7/site-packages/ray/experimental/state/state_manager.py", line 80, in StateDataSourceClient
    def __init__(self, gcs_channel: grpc.aio.Channel):
AttributeError: module 'grpc' has no attribute 'aio'

• Then step ray submit config.yaml script.py crashes with:

ConnectionError: Could not find any running Ray instance. Please specify the one to connect to by setting `--address` flag or `RAY_ADDRESS` environment variable.

• When I tried the same in a new cluster I got another error [2022-04-28 08:44:54,724 E 2087 2087] core_worker.cc:137: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

Looks like either a bug in Ray, or the documentation is not up to date, or is not beginner-friendly (I've never used Ray before). What's going on here? Maybe is there any more beginner-friendly tutorial how to use Ray on AWS?

Answer 1

I had the same error on an HPC using Slurm. For me specifying another port than the default one worked.