CODEWITHSUNDEEP
python
Pavlin
Pavlin

Reputation: 5528

Python ImportError on Python 3.6, 3.7, 3.8, but not on 3.9

I have come across a bizarre import error on my CI that I can't reproduce locally, and have no idea how to tackle. I use Azure pipelines for my CI and run a build matrix to build wheels for my package. Everything is built as expected. Today, I wanted to release a new version and I changed the code a little bit. My file structure is

openTSNE/
   __init__.py
   utils.py
tests/
   test_tsne.py
setup.py

so the source is in openTSNE and the tests are in the tests folder. I attempt to import the function is_package_installed in test_tsne.py from openTSNE.utils like so

from openTSNE.utils import is_package_installed

I build the wheels for all versions of Python 3.6 - 3.9, on Mac, Windows, and Linux. I also build an sdist on Linux. I first build the packages on each CI VM, install it, and run my unittest suite on the installed package.

That's the setup. Here's the absolutely bizarre part: Everything works on my wheels OSX-py39, Win-py39, and all linux wheels. It does NOT work on the Linux sdist version, or on OSX-py36 through py38, or on Win-py36 through py38. See the CI build log https://dev.azure.com/pavlingp/openTSNE/_build/results?buildId=359&view=logs&j=9adead4c-7271-5cd3-b143-2c28e08460f5&t=9572adb2-9d5f-5130-5c60-5fd15aeb658c

The tests fail with

tests/test_tsne.py:19: in <module>
from openTSNE.utils import is_package_installed
ImportError: cannot import name 'is_package_installed'

The source code can be found on GH. The function and the offending test. I have tried to reproduce this on my local MacBook without success.

I am at a loss of what could be going wrong. This is also the first time this has happened, I've been working on this project for a while without any such issues. Any help or pointer in the right direction, or even how I could approach debugging/reproducing this would be greatly appreaciated.

Upvotes: 1

Views: 1072

Answers (2)

Pavlin
Pavlin

Reputation: 5528

I have tracked down the issue. When I was running the CI builds, I hadn't bumped the version. The live version on PyPI is 0.4.4, and I was working on releasing v0.5.0. However, I hadn't bumped the version in the package, so when I was building wheels and the sdist package, they were being tagged with 0.4.4.

Now, I had tried to get around this by running pip with the --find-links switch

pip install --user --force-reinstall --find-links dist openTSNE

I was, however, not aware that pip will always prioritize packages on the PyPI over local binaries or sdists. So, on my CI servers, I was building the wheels, but when I actually installed them to run the unit tests, pip was pulling down 0.4.4 from PyPI and not using the newly built package. This is why the import error was occurring -- I had only added the offending is_package_installed function in this version, and it didn't exist in 0.4.4.

Bumping the version fixed the problem, as pip wasn't able to find 0.5.0 on PyPI and was forced to use the local binary.

It is bizarre to me that it doesn't behave in the same way on all platforms, yet here we are.

Upvotes: 2

Richard Taujenis
Richard Taujenis

Reputation: 64

Did you create a init.py file?

openTSNE/
utils.py/
__init__.py
tests/
test_tsne.py
setup.py

If you remove the init.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.

The init.py file is usually empty, but can be used to export selected portions of the package under more convenient name, hold convenience functions, etc.

Upvotes: 0

Related Questions