CODEWITHSUNDEEP
pythonperformancescipy
user541686
user541686

Reputation: 210525

Why does importing some SciPy modules take so long on Windows?

On some of my otherwise-relatively-fast Windows machines, importing SciPy modules like scipy.signal takes an absurdly long time (over a second).

And I can't simply Ctrl-C in the middle to see what's going on, thanks to good old SciPy overriding the stack trace:

forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source
KERNELBASE.dll     00007FFBD9AFD37F  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFBDBE213D2  Unknown               Unknown  Unknown
ntdll.dll          00007FFBDC8C54E4  Unknown               Unknown  Unknown

This doesn't happen on all my machines, though. What's going on?

Upvotes: 1

Views: 302

Answers (1)

user541686
user541686

Reputation: 210525

This was a very frustrating issue for me to track down, so I'm posting the solution here.

I ended up using Visual Studio's debugger to break into the program and see what it's doing.

It turns out there are 2 orthogonal factors that contribute to this:

  • For reasons beyond my comprehension, some package(s) (I'm looking at scikits.odes here) have __init__.py's that have this line:

    __import__('pkg_resources').declare_namespace(__name__)

    For reasons beyond my comprehension, once scikits.odes is imported (which SciPy does for some bizarre reason), this line ends up running, loading an ungodly number of packages from your installation (seemingly, most if not all of them). This grows slower the bigger your installation.

    The solution to this problem is to either uninstall this offending package, or if you really need it, to comment out this line (or do something equivalent) and hope it doesn't get offended.

  • You haven't compiled your .py files to .pyc (run python -m compileall . inside your Python installation directory). This actually helps for a lot of packages. (I was skeptical at first.)

Fixing both of these issues reduced my import time to around 0.3 seconds, which seems normal.

Upvotes: 6

Related Questions