CODEWITHSUNDEEP
modulejuliadistributed-computing
cako
cako

Reputation: 294

How do you load a module @everywhere inside a function in Julia

I am trying to load a module in my workers after creating them with addprocs. When addprocs is called at the top level, everything is fine. However, I am not able to do the same thing when I wrap the code in a function.

In my case, I am adding workers dynamically, so it is not feasible to call @everywhere using XXX always on the top level, I would need to do this inside a function.

In short, this works:

addprocs(1)
@everywhere using XXX

and this doesn't:

function myaddprocs()
    addprocs(1)
    @everywhere using XXX
end

Any ideas?

Upvotes: 4

Views: 1706

Answers (3)

L. Plagne
L. Plagne

Reputation: 23

Cako's solution is helpful but I had to add the module as first argument to macroexpand :


eval(macroexpand(Distributed,quote @everywhere using DistributedArrays end))

Upvotes: 1

cako
cako

Reputation: 294

After a bit more investigation, I have pinpointed some of the issues which were making my code not work.

  1. Imports must happen after addprocs. If an import has happened before, the import must be prefixed by an @everywhere.

  2. Top-level expressions (such as using) inside functions do not work, unless wrapped in an eval statement.

A fix to my code would be:

function myaddprocs()
    addprocs(1)
    eval(macroexpand(quote @everywhere using XXX end))
end

Examples

I have tested the following snippets on Julia 0.6.1. I have also tested them using the same version on a SGE cluster (OGS/GE 2011.11p1), by substituting all addprocs by addprocs_sge, and importing ClusterManagers.jl. The following snippets work:

  • using after addprocs:

    addprocs(1)
using SpecialFunctions
pmap(x->SpecialFunctions.sinint(1), workers())

  • using before and after addprocs, the second with @everywhere:

    using SpecialFunctions
addprocs(1)
@everywhere using SpecialFunctions
pmap(x->sinint(1), workers())

  • using wrapped in eval after addprocs within function

    function getprocs()
    addprocs(1)
    eval(Expr(:using,:SpecialFunctions))
    pmap(x->SpecialFunctions.sinint(1), workers())
end
getprocs()

  • Same as before with with @everywhere applied to eval

    function getprocs()
    addprocs(1)
    @everywhere eval(Expr(:using,:SpecialFunctions))
    pmap(x->sinint(1), workers())
end
getprocs()

  • Same as before with with @everywhere within eval instead

    function getprocs()
    addprocs(1)
    eval(macroexpand(quote @everywhere using SpecialFunctions end))
    pmap(x->sinint(1), workers())
end
getprocs()

These snippets, on the other hand, do not work:

  • using before addprocs

    using SpecialFunctions
addprocs(1)
pmap(x->SpecialFunctions.sinint(1), workers())

  • using before and after addprocs

    using SpecialFunctions
addprocs(1)
using SpecialFunctions
pmap(x->SpecialFunctions.sinint(1), workers())

  • using within function

    using SpecialFunctions
function getprocs()
    addprocs(1)
    @everywhere using SpecialFunctions
    pmap(x->sinint(1), workers())
end
getprocs()

  • using within function

    function getprocs()
    addprocs(1)
    using SpecialFunctions
    pmap(x->SpecialFunctions.sinint(1), workers())
end
getprocs()

Upvotes: 7

Florian Oswald
Florian Oswald

Reputation: 5134

you don't need @everywhere. this works for me:

addprocs()
using mymodule  # loads code on all procs but brings in to scope only on master process.

this is what you want if you want to then do pmap(x->fun(x),workers()), and fun is exported from mymodule. you can read this up here: https://docs.julialang.org/en/release-0.6/manual/parallel-computing/#Code-Availability-and-Loading-Packages-1

Upvotes: 2

Related Questions