CODEWITHSUNDEEP
python-3.xlinuxpython-import.so
Bernd Wechner
Bernd Wechner

Reputation: 2143

ModuleNotFoundError even though module is in sys.path

I am puzzled by an apparent inconsistency here. It relates to the setproctitle module. Basically python3.6 can see it, and python3.8 can't.

I can demonstrate with a couple of simple command line demos running:

python3.6 -c "import sys; print(sys.path); import setproctitle; print(setproctitle.__file__)"

and

python3.8 -c "import sys; print(sys.path); import setproctitle; print(setproctitle.__file__)"

And here is the comparison:

$ python3.6 -c "import sys; print(sys.path); import setproctitle; print(setproctitle.__file__)"
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/bernd/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.6/dist-packages']
/usr/lib/python3/dist-packages/setproctitle.cpython-36m-x86_64-linux-gnu.so
$ python3.8 -c "import sys; print(sys.path); import setproctitle; print(setproctitle.__file__)"
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/bernd/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setproctitle'

The salient observation being that setproctitle is found in /usr/lib/python3/dist-packages and that directory is in sys.path in both cases, for Python3.6 and Python3.8.

What can we conclude from that?

Given the loaded file named /usr/lib/python3/dist-packages/setproctitle.cpython-36m-x86_64-linux-gnu.so has a suspicious 36m in its name it suggests that Python3.8 might be looking for /usr/lib/python3/dist-packages/setproctitle.cpython-38m-x86_64-linux-gnu.so, but that would raise the ancillary question: How do we (canonically) get the 3.8 package of setproctitle?

I'll admit I tried sneakily:

sudo ln /usr/lib/python3/dist-packages/setproctitle.cpython-36m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/setproctitle.cpython-38m-x86_64-linux-gnu.so

and tried again but there's more to it than just the name. I have tried finding information on this on-line but am a little stuck having found nothing so far. If there are some simple summaries as to what is going on here and how to have seemingly binary libraries side by side for different Python versions, I'd be pleased for pointers.

My guesses are:

  1. The .so fileis a binary file that contains links to a specific Python version
  2. It would thus need to be rebuilt for Python3.8
  3. It may be, given the youth of Python3.8 that it has not yet been rebuilt for distribution
  4. It might be possible to rebuild it on the fly, perhaps with apt if one new the syntax.

Upvotes: 1

Views: 2076

Answers (1)

wim
wim

Reputation: 363063

You have an extension module, which was built and tagged with a CPython 3.6 / Linux runtime. Simply renaming that file is not a good idea and is unlikely to work. Even though that file is present on the sys.path, it will be ignored by a Python 3.8 runtime since the tag is not matching.

See PEP 3149 -- ABI version tagged .so files.

To get a 3.8 version, you should build - i.e. install from source - using a 3.8 runtime:

python3.8 -m pip install setproctitle

Upvotes: 2

Related Questions