CODEWITHSUNDEEP
pythoncythonsetuptoolspython-wheel
Pavlin
Pavlin

Reputation: 5528

How to test Python wheels on CI

I'm am running my tests on Azure Pipelines (but the same thing applies to Travis and Appveyor). I have a Python package, let's call it calculator which contains cython extensions. When I push to my repository, the CI servers clone my package and install it using python setup.py develop, then run pytest to run the tests. Running python setup.py develop builds the extensions inplace, so the shared objects are in the actual directory. So when my unittests run import calculator, the cwd is the source directory, so the actual copies of the python scripts are used, which contain the shared object files right there, so everything works fine.

Now, I want to replace python setup.py develop with python setup.py install. Now, the extensions are no longer built in the current source directory, but are packaged and installed into the python path. However, when pytest is called after installation, python can still find the calculator module in the working directory, and (I guess) ignores the properly installed calculator module. And because the extensions weren't built inplace, the tests fails saying the extension module is not found.

The reason I want to do this is because I also have wheels set up to be built on the CI servers. When I build a wheel, I would also like to make sure nothing went wrong and install that wheel and run tests on it. Similarily to python setup.py install, I suppose the calcualtor module in the working directory takes precedence over my installed wheel, and results in extension import errors.

How do people typically deal with this?

Upvotes: 2

Views: 1072

Answers (1)

Pavlin
Pavlin

Reputation: 5528

I've read up quite a bit on this since posting this question and the problem stems from the fact that Python automatically prepends the working directory to sys.path. Since CI servers have their working path set to the source directory, the actual source files take precedence over the installed library. There are two solutions:

  1. We can switch the package over to a src layout, which just means renaming the typical convention of my calculator package containing source files from calculator to src. This requires that setup is then properly adjusted, and that the src directory contains a single directory called calculator. This, in my opinion, is ugly, so I went with the much simpler option
  2. simply remove or rename the source directory on the CI server, so that the package can't be found in the working directory. In my specific case, I just ran mv calculator src before running my tests.

Upvotes: 2

Related Questions