CODEWITHSUNDEEP
pythonimportgarbage-collectionpython-internals
Ben
Ben

Reputation: 7124

Does python sometimes re-import modules during runtime?

I have a python codebase I'm using for research. The codebase imports libraries like numpy and pytorch, but also some custom tooling (i.e. other python packages I've written and want to use). These custom packages are installed using pip install -e . into a virtual environment.

My workflow is such that I will launch a long-running job (a week or so) and then continue to modify or refactor the codebase in parallel. I'm becoming increasingly suspicious (paranoid?) that some (not all) of these modifications are changing runtime behavior.

Unfortunately, I have not been able to isolate this into a concrete example. Instead, I feel like python is gaslighting me with unexplainable results.

Is there something with python's garbage collector and working with editable installs where some modules are reloaded? Or maybe all modules are not loaded upfront?

I have explicitly seen this behavior when making large changes: Such as

 - run "experiment" script
    - "experiment" script imports package 'tools'
 - While experiment script is running, update `tools` from v1.0 to v2.0
 - Assert checker in "experiment" script that checks 
   tools.__version__ == v1.0 causes code to crash

Upvotes: 2

Views: 189

Answers (1)

Thom Marchesini
Thom Marchesini

Reputation: 61

I think your bold statement already has the answer in it.

What might be happening is that your editable flag "-e" could cause some issues. But as far as I understand it this should not happen during runtime. However, if the python kernel is restarted in the script somewhere it would lead to issues.

A solution would then be to not use the -e flag when installing the packages in the virtual environment where your script runs (e.g. python setup.py install). This is also preferable to do if you do not want it to implicitly update the versions it is running on.

Upvotes: 1

Related Questions