CODEWITHSUNDEEP
filejulialambdifypackagecompiler.jlsymengine
Zaph
Zaph

Reputation: 111

How to save a lambdifyed expression to a file and load it in a new Session?

My plan is to first calculate allot of expressions symbolically, safe them, then use them later in several extensive Metropolis Monte-Carlo-simulations. While I initially tried Sympy and then Symbolics, I am now intending to use SymEngine as it just appears to be significantly faster at all relevant steps. When using SymEngine a big part of the time used to generate the functions is actually not the symbolic computations but the lambdifying. Therefore I wish to save the functions generated by lambdify and not the symbolic expressions. However I seem to be incapable of finding a way to successfulyl load them into a new Session.

I initially tried Using BSON as proposed here: https://discourse.julialang.org/t/is-there-a-way-to-write-a-function-into-jld-or-other-h5-file/77718/2. But sadly reloading only works in the same Session. This can for example be reproduced by doing:

using SymEngine
using BSON: @save, @load

function mwe()
    @vars X
    f = X
    return(lambdify(f,[X]))
end

f = mwe()

@save "test.bson" f

then (in a new Session):

using SymEngine
using BSON: @save, @load

function mwe()
    @vars X
    f = X
    return(lambdify(f,[X]))
end

@load "test.bson" f
println(f(1))
f = "nofunction"
@load "test.bson" f
println(f(1))

If I do both in the same Session the output is

1       
1

as expected.

However if I run the second part in a new Session I get:

ERROR: UndefVarError: ###312 not defined

I also tried a naive (maybe there are more elaborate ways of applying these?) aproach of using Serialization, FileIO as well as JLD2. All of them resulting in similar behavior working fine if loaded in same session, but either erroring out or warning and then behaving unexpectedly when loading in a different Session.

What did work is using Symbolics write("test.jl", string(f)) as described here: https://discourse.julialang.org/t/using-serialization-to-store-then-load-a-lambdified-function/85884/2. However since Symbolics is much slower then SymEnginge this does not really seem to be the ideal way to go to me.

Update: I tried to find a work around by generating a sysimage with PackageCompiler. The Idea was to create a module in which all the symbolic calculations are done then a constant containing a list of all the lamdifyed expressions is created. Then create a sysimage of that module and hope that I could use the lambdified expression in a new Session when started using the sysimage. If I do not generate the constant, creating the sysimage works fine. But once I add the lines that should create the list of functions

const Bookofterms = generate_terms(3,3)

I get a somewhat lengthy error message containing:

ERROR: The following 1 direct dependency failed to precompile:
TimeInt [14fa5fd4-ce9e-41c9-8e49-66abaaa3fbcd]

(TimeInt is the custom package)

Failed to precompile TimeInt [14fa5fd4-ce9e-41c9-8e49-66abaaa3fbcd] to C:\Users\...
ERROR: LoadError: ArgumentError: Expression does not lambdify

caused by: Evaluation into the closed module `SymEngine` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `SymEngine` with `eval` during precompilation - don't do this.

I am honestly quite confused on whats going on.

Update: Since did not get any responses after a few days I decided to try copying this post to: https://discourse.julialang.org/t/how-to-save-a-lambdifyed-by-symengine-expression-to-a-file-and-load-it-in-a-new-session/111165?u=zaph

Upvotes: 1

Views: 152

Answers (1)

Zaph
Zaph

Reputation: 111

During further investigation I found out some things: First of all: Saving a Julia function defined in a way like

function f(x,y,z)
    do stuff
    return something
end

and then loading it in another session doesn’t work with BSON and I assume it does not with any of the Methods I tried before either. So no need to use lambdify to produce such errors.

However the Code example given by https://discourse.julialang.org/t/is-there-a-way-to-write-a-function-into-jld-or-other-h5-file/77718/2 still does work. So it appears to be the case that the Problem lies somewhere in the function name as anonymous functions can be saved.

What I tried then is using the following:

function lambdify(ex, vars = [])
    body = convert(Expr, ex)
    if isempty(vars)
        syms = Symbol.(free_symbols(ex))
    else
        syms = Symbol.(vars)
    end
    fn = eval(:($(Expr(:tuple,syms...)) -> $body));
    return(fn)
end

instead of lambdify.

And indeed this enabled me to save the resulting functions and load them in a new Session using BSON.

Now finally it should be noted that loading with BSON the way I did was very slow, in fact even slower then recalculating the functions. However that approach also enabled the usage of Serialization and this does indeed provide a serious speedup compared to recalculating in every new Session.

So the answer to my Problem is using the above defined function instead of lambdify and just saving and loading with Serialization.

Edit: This appears to not be the entire truth, as when I try to put the above function into a module and use the function from the module, to genereate the lambidfied expressions, then serialzing and deserialzing in new Session does not work again. I asked a new question regarding that specifc behavior here.

Upvotes: 0

Related Questions