CODEWITHSUNDEEP
python
Takkun
Takkun

Reputation: 6361

What's causing this delay when importing libraries?

I'm seeing an issue where the execution of my python script can sometimes take longer to import libraries. This happens if I have not executed the script recently or if I run the script from a different server. After the first delay, the import times become much faster. I was wondering what is causing this slow import and if there is any way to prevent it?

import time
s_time = time.time()
import sys,re,os,logging,signal
from argparse  import ArgumentParser
print('Internal Import Time: {}'.format(time.time() - s_time))
s_time = time.time()
from backtrace import Backtrace,BacktraceSet
from report    import Report
from core      import Core
from burtapi   import BurtAPI
print('External Import Time: {}'.format(time.time() - s_time))

In this example backtrace,report,core, and burtapi are libraries I created.

[name@server1 tool]$ ./tool --python
Internal Import Time: 2.8281359672546387
External Import Time: 13.053943157196045
Enter/Paste your content. Ctrl-D to save it.
^CYou pressed Ctrl+C!
[name@server1 tool]$ ./tool --python
Internal Import Time: 0.12279081344604492
External Import Time: 0.6948020458221436
Enter/Paste your content. Ctrl-D to save it.
^CYou pressed Ctrl+C!

ssh to different server (with same storage mount)

[name@server2 tool]$ ./tool --python
Internal Import Time: 3.0217390060424805
External Import Time: 13.151482105255127
Enter/Paste your content. Ctrl-D to save it.
^CYou pressed Ctrl+C!
[name@server2 tool]$

./tool is a bash script that calls python3 /path/to/script.py

Upvotes: 1

Views: 729

Answers (1)

Alexander Ejbekov
Alexander Ejbekov

Reputation: 5940

There are more factors here but there are three major ones:

First - importing a module requires finding the modules on your system, which requires reading data from your disk.

Second - with that, if there are any changes done inside those modules, the interpreter would need to bite-compile them.

Last, but not least, when the interpreter starts importing them, the modules themselves might contain instructions/code that needs to be executed which could be performing a lot of operations. Say imagine one of those modules connects to a database or has to filter through a large hashtable...

Edit: I should point out that those are assumptions and it's hard to say what exactly is happening in your case but in most cases this is what's happening.

Upvotes: 1

Related Questions