CODEWITHSUNDEEP
julia
Sameeresque
Sameeresque

Reputation: 2602

Module does not support precompilation but is imported by a module that does

I'm working with pmap and Distributed arrays in Julia in order to parallelize a code. When the command @everywhere include("./code.jl") is run, a long list of warning messages is outputted, and in the process consumes a considerable amount of time. The packages that are inside the file code.jl are, for example:

using DataFrames
using Images

I'm not sure how to address the problem of missing from the cache

This is the line that is run

@everywhere include("./code.jl")

I did not expect a litany of complaints. However, the output is similar to this on all the workers:

From worker 2:  │ This may mean CategoricalArrays [324d7699-5711-5eae-9e2f-1d82baa6b597] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:947
      From worker 6:    ┌ Warning: Module CategoricalArrays with build ID 5344443537363826 is missing from the cache.
From worker 4:  ┌ Warning: Module DataFrames with build ID 5344463834994296 is missing from the cache.
      From worker 4:    │ This may mean DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:947
      From worker 2:    ┌ Warning: Module DataFrames with build ID 5344464953933087 is missing from the cache.
      From worker 2:    │ This may mean DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] does not support precompilation but is imported by a module that does.

Upvotes: 3

Views: 2419

Answers (2)

Przemyslaw Szufel
Przemyslaw Szufel

Reputation: 42194

While @tholy answer is right - you will need to remember to recompile every time you update your packages.

Another approach to fight the race condition is to import the packages locally and than in distributed version:

using DataFrames
@everywhere using DataFrames

Assuming that your file only imports packages and defines functions you could try:

include("./code.jl")
@everywhere include("./code.jl")

This should help and you do not need to remember to recompile when you update some of the packages used by your application.

Upvotes: 4

tholy
tholy

Reputation: 12179

You may have a package that needs recompilation, and you're getting a race condition in which all workers decide they need to independently recompile it. Try starting a single julia session and typing ]precompile (the ] for entering pkg mode). Once that finishes, for safety you could try using DataFrames just to make sure it worked. Then try your code above.

Upvotes: 6

Related Questions