CODEWITHSUNDEEP
windowsparallel-processingjulia
Petr Hlavenka
Petr Hlavenka

Reputation: 21

Issues with parallel Julia code on windows cluster

I'm setting up a small windows cluster for parallel speedup of my Julia code (2x32 cores).

I have following questions:

  1. Is there a way to suppress loading of a module (e.g. "using PyPlot") on the remote machines? In my code, I use my workstation for initialization and data presentation, whereas the cluster is used for heavy calculation without any need for PyPlot, Dataframes etc.

  2. This code loading on the remote machines is even more annoying as the PyPlot (and any other package) fails to populate help database by giving following error message: (actually a lot of errors from every worker)

    exception on : 1: 1ERROR: opening file C:\Users\phlavenk\AppData\Local\Julia-0.3.6\bin/../share/julia\helpdb.jl: No such file or directory

Running on Julia 3.6/ x64 / Windows7, identical directory structure and versions everywhere.

My addprocs command is following:

addprocs(machines, 
  sshflags=`-i c:\\cygwin64\\home\\phlavenk\\.ssh\\id_rsa`, 
  dir=`/cygdrive/c/Users/phlavenk/AppData/Local/Julia-0.3.6/bin`, 
  tunnel=true)

Thank you very much for your advice

Upvotes: 2

Views: 240

Answers (2)

Greg
Greg

Reputation: 91

You could sequence the statements in this order:

using PyPlot
using ModuleNeededOnMasterProcessOnly
addprocs(...)
using ModuleNeededOnAllProcesses

Upvotes: 0

mbeltagy
mbeltagy

Reputation: 181

"using" causes a module to be loaded on all the processes. To load a module on a specific machine you use "include". e.g.

if myid()==1 include("/home/user/.julia/PyPlot/src/PyPlot.jl") end

You can then do your plotting by PyPlot.plot(...) on your local machine.

Upvotes: 1

Related Questions