CODEWITHSUNDEEP
rparallel-processingshared-memory
Rappster
Rappster

Reputation: 13080

Shared memory for parallel processes

It's been a while since the last time I looked (e.g. outdated nws package) and so I wondered if anything "happened" in the meantime.

Is there a way to share memory across parallel processes?

I would like each process to have access to an environment object that plays the role of a meta object.

Upvotes: 2

Views: 1318

Answers (3)

Adam Ryczkowski
Adam Ryczkowski

Reputation: 8064

You can do this efficiently via a new yaplr package.

First install it 

devtools::install_github('adamryczkowski/yaplr')

R session number 1:

library(yaplr)
send_object(obj=1:10, tag='myobject')
# Server process spawned

R session number 2:

library(yaplr)
list_objects()
#          size                    ctime
# myobject   62 Sat Sep 24 13:01:57 2016
retrieve_object(tag='myobject')
# [1]  1  2  3  4  5  6  7  8  9 10
remove_object('myobject')
quit_server()

The package uses bigmemory::big.matrix for efficient data transfer: when copying a big object between R processes no unnecessary copies are made: only one serialization and one unserialization.

No network sockets are used.

Upvotes: 1

Adam Ryczkowski
Adam Ryczkowski

Reputation: 8064

What will work quite efficiently is a shared matrix via the bigmemory package. You can serialize/unserialize pretty every R object into such matrix.

Unfortunately the only way you can share the matrices between the processes is via their descriptors, which are not deterministic (i.e. unless you communicate with the other process, you can not get a descriptor). To solve this chicken-and-egg problem you can save the descriptor on the chosen location on the filesystem. (The descriptor is really small, the only non-trivial thing it contains is a memory address of the actual bigmatrx).

If you are still interested, I can post the R code.

Upvotes: 1

Steve Weston
Steve Weston

Reputation: 19667

The rredis package provides functionality that is similar to nws. You could use rredis with the foreach and doRedis packages, or with any other parallel programming package, such as parallel.

Upvotes: 2

Related Questions