CODEWITHSUNDEEP
pythonredismultiprocessingshared-memoryray
Jwf
Jwf

Reputation: 195

Python Ray shared memory access

I have a question concerning the access of the shared memory within the Ray Framework.
Imagine the following setup on 1 machine:

  1. Start Ray cluster
  2. Start a process/worker python script w1.py, which puts an object O1 into the shared memory via ray.put(O1)
  3. Start a process/worker python script w2.py, which tries to get O1 from the shared memory via ray.get(...)

Is there a way to access the object O1 (put into shared memory from w1.py process) from another worker process w2.py?

When I execute ray.objects() from w2.py, I get the obejct reference string, but how could I retrieve the object from the shared memory then? I can not init a ObjectRef object in w2.py

Upvotes: 1

Views: 1209

Answers (1)

Sang
Sang

Reputation: 925

This is not natively supported. The reason is ray’s objects have various metadata that is for various features (ex, perf optimization or automatic memory management using reference counting).

If you’d like to achieve this, I think there are 2 solutions.

  1. Use a detached actor api. Detached actors are actors whose lifetime is not fare-sharing with drivers. Once you create a detached actor, you can obtain the actor handle using ray.get_actor API. This way, you can put an object inside a detached actor and access from multiple drivers.

  2. There’s another way which uses cloudpickle, but I am not so familiar with this solution, so I won’t write about it. Please go to Ray’s discussion page in its Github repo to ask more details about it.

Upvotes: 1

Related Questions