CODEWITHSUNDEEP
python
YGA
YGA

Reputation: 10010

Speeding up the python "import" loader

I'm getting seriously frustrated at how slow python startup is. Just importing more or less basic modules takes a second, since python runs down the sys.path looking for matching files (and generating 4 stat() calls - ["foo", "foo.py", "foo.pyc", "foo.so"] - for each check). For a complicated project environment, with tons of different directories, this can take around 5 seconds -- all to run a script that might fail instantly.

Do folks have suggestions for how to speed up this process? For instance, one hack I've seen is to set the LD_PRELOAD_32 environment variable to a library that caches the result of ENOENT calls (e.g. failed stat() calls) between runs. Of course, this has all sorts of problems (potentially confusing non-python programs, negative caching, etc.).

Upvotes: 25

Views: 15836

Answers (5)

Nico Schlömer
Nico Schlömer

Reputation: 58791

When trying to speed things up, profiling is key. Otherwise, how will you know which parts of your code are really the slow ones?

A while ago, I've created the runtime and import profile visualizer tuna, and I think it may be useful here. Simply create an import profile (with Python 3.7+) and run tuna on it:

python3.7 -X importtime -c "import scipy" 2> scipy.log
tuna scipy.log

enter image description here

Upvotes: 4

Chris B.
Chris B.

Reputation: 90231

Something's missing from your premise--I've never seen some "more-or-less" basic modules take over a second to import, and I'm not running Python on what I would call cutting-edge hardware. Either you're running on some seriously old hardware, or you're running on an overloaded machine, or either your OS or Python installation is broken in some way. Or you're not really importing "basic" modules.

If it's any of the first three issues, you need to look at the root problem for a solution. If it's the last, we really need to know what the specific packages are to be of any help.

Upvotes: 0

BrainCore
BrainCore

Reputation: 5412

If you run out of options, you can create a ramdisk to store your python packages. A ramdisk appears as a directory in your file system, but will actually be mapped directly to your computer's RAM. Here are some instructions for Linux/Redhat.

Beware: A ramdisk is volatile, so you'll also need to keep a backup of your files on your regular hard drive, otherwise you'll lose your data when your computer shuts down.

Upvotes: 2

Alex Martelli
Alex Martelli

Reputation: 881765

zipping up as many pyc files as feasible (with proper directory structure for packages), and putting that zipfile as the very first entry in sys.path (on the best available local disk, ideally) can speed up startup times a lot.

Upvotes: 11

Seth
Seth

Reputation: 46433

The first things that come to mind are:

  • Try a smaller path
  • Make sure your modules are pyc's so they'll load faster
  • Make sure you don't double import, or import too much

Other than that, are you sure that the disk operations are what's bogging you down? Is your disk/operating system really busy or old and slow?

Maybe a defrag is in order?

Upvotes: 5

Related Questions