CODEWITHSUNDEEP
julia
Cokes
Cokes

Reputation: 4083

Parallel computing in Julia and mis-allocating # of cores

I used the pmap() function in Julia to write a parallel code.

I then secured four cores on the cluster and ran a script:

julia -p 12 my_parallel_program.jl

Should I now cancel my job? What exactly is happening now that I told Julia there are 12 cores when there are really 4? Will it run just as fast as if I had julia -p 4 my_parallel_program.jl?

Upvotes: 2

Views: 985

Answers (2)

Przemyslaw Szufel
Przemyslaw Szufel

Reputation: 42204

The option -p 12 launches a total 13 parallel processes (one master process plus 12 slave/worker processes). Depending on your problem setting this might be desired or not.

The rule of thumb is to launch a number of slave processes that matches the number of cores on your machine (this can be checked via nproc bash command).

However, in scenarios that are strongly dependent on some external processing (e.g. waiting for network IO), creating a larger number of processes that the number of cores might completely make sense. On the other hand, in compute intensive cases (such as numerical simulations) the number of slave processes should be equal to the number of available cores. Last, but not least, please note that if you run your a compute intensive process on laptop you can easily cause it to overheat (theoretically it should not happen but I managed to melt-down 3 laptop CPUs this way). Hence, if it is numerical simulation on laptop - use no more than 75% of available cores.

Having that said please note that there are many alternatives to multiprocessing in Julia:

  • green threads with @async - those are great for network intensive scenarios such as web-scraping
  • multi-threading with Threads.@threads - this requires setting the JULIA_NUM_THREADS system variable. This option allows to share common data across the threads. The advantage is that are no communication issues between process (compared to -p), the disadvantage is that the utilization of locking mechanism is required to prevent to threads to change the same piece of data simultaneously.
  • multiprocessing (the one with -p option that you used). Here Julia spawns the given number of slave processes. If you need to communicate across processes you should use the ParallelDataTransfer.jl package
  • distributed multiprocessing (using the --machinefile Julia launch option). This is very powerful because it allows to run -p-style code across huge clusters of machines running Julia. All what is needed to configure this is passwordless SSH and opening TCP/IP network connections across machines in the cluster.

The optimal choice depends on your computational scenario but some hints have been given above.

Upvotes: 7

Bogumił Kamiński
Bogumił Kamiński

Reputation: 69829

You told Julia to start 12 worker processes and Julia obeyed. You could check it calling nworkers function. The number of logical cores Julia sees can be checked by Base.Sys.CPU_CORES.

In general starting more processes than cores most likely will degrade the performance of computations as 12 processes compete for time of 4 cores.

Here is an example session to confirm this (run on machine with 4 logical cores):

$ julia -p 12
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0-alpha.0 (2018-05-31 00:07 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> nworkers()
12

julia> Base.Sys.CPU_CORES
4

Upvotes: 2

Related Questions