CODEWITHSUNDEEP
rparallel-processingr-package
Jacob Socolar
Jacob Socolar

Reputation: 1202

R package design: how to export internal functions to a cluster

I'm working on an R package, and I need to run a function myfun on a cluster using parallel::parLapply. myfun calls several additional functions from my package, which in turn call more functions, some of which have multiple methods... so passing all of the functions and methods to the cluster explicitly by name is very cumbersome.

The standard advice, as I understand it, is to run parallel::clusterEvalQ({library("my_package")}). But the call to library("my_package") is apparently anathema to R-CMD-check. And I have reason to believe that my-package:::function won't fly on CRAN either.

What is the standard approach here? Do I need to export every single relevant function and method by name?

Upvotes: 5

Views: 447

Answers (1)

Jacob Socolar
Jacob Socolar

Reputation: 1202

Ok, this seems to work, (it passes R-CMD-check on GitHub):

parallel::clusterExport(cl = cl, 
                        unclass(lsf.str(envir = asNamespace("my_package"), 
                                        all = T)),
                        envir = as.environment(asNamespace("my_package"))
                        )

Hope it's useful to others.

There's also probably a nifty solution available via the globals package, but I haven't been able to get this to pass checks on GitHub actions.

Upvotes: 2

Related Questions