CODEWITHSUNDEEP
importmoduleironpython
GradlonGwen
GradlonGwen

Reputation: 13

Import module in IronPython 2.7.1 very slow

(first question on StackOverflow, glad to be there :))

I am using IronPython 2.7.1 and C# .net 4.0.

I use C# to launch my python script. I have about 20 personal modules that are imported a lots of time. E.g :

If I have module1.py, module2.py, module3.py, module4.py and main_script.py.

main_script.py imports module1 and module2

Both module1 and module2 import module3.

module1 and module3 import module4

etc.

Modules can have a large amount of code lines.

What I see is when I execute my main_script.py, it takes about 4-5 sec to just import modules.

I tried to use pyc.py to compile all my modules in a dll, and then used ngen on it, but I saw no differences when adding this dll using myEngine.Runtime.LoadAssembly().

Then I wanted to use py_compile.py to get the pyc files, but is seems not working as the IronPython.Runtime.FunctionCode type is not supported in the IronPython.Modules.MarshalWriter class (function WriteObject(object o). (I got "unmarshallable object" exception when trying to compile).

I am not very familiar with Python nor IronPython, and maybe I did not understood all the subtleties of the language (I think so, actually). I was searching the net for a solution, but it seems I am stuck right now.

Any idea to improve the import performance ?

Upvotes: 1

Views: 1410

Answers (1)

Jeff Hardy
Jeff Hardy

Reputation: 7662

Taking 4-5 seconds to do imports, especially for large modules, is not unexpected for IronPython 2.7.1. I would pyc.py to improve it, but I also think that it isn't as useful as it once was - IronPython's imports are a lot faster than they used to be, so pyc.py is less useful.

The thing is, IronPython does a lot more than Python does when it imports a module[1]. Python has to parse it and produce bytecode which it then executes. IronPython has to produce DLR trees which are then converted to interpreter instructions - and possibly also IL if they trip the compilation limit, which means running the .NET JIT to produce machine code.

All of that work is wasted if the script only takes a few seconds to run; IronPython is better for long-running processes. However, the short Python script is extremely common, and IronPython is extremely poor for those sorts of scripts.

There are two ways we're working at solving this, one of which you alluded to. Work is being done to support standard .pyc files with an interpreter optimized for startup time but not throughput - short scripts will benefit, but long-running code will suffer. Second, porting IronPython to mobile platforms requires disabling dynamic code generation, so making the DLR interpreter fast will be very important; this work will make uncompiled code faster to start as well.

The one thing we cannot overcome is the fact that .NET processes generally take longer to start than plain C ones. That overhead can be reduced, but it requires some fairly deep optimization that probably won't be done for a while.

[1] Python's import process is so fast that the stat calls to find the file are much greater than the time to parse & compile it.

Upvotes: 3

Related Questions