Compiled python version: ModuleNotFoundError at import

Question

I have a project in a conda environment that runs with python 3.7.7 (on linux). When I recompile the same version of python (3.7.7) and put/replace the executable at the same location, I expect the program to run the same way, but the import fails.

With the original version of python:

(condaenv) mypc:~/Proj$ /home/me/.conda/envs/condaenv/bin/python3.7.bak
Python 3.7.7 (default, Mar 26 2020, 15:48:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gym
>>> quit()

With the compiled version:

(condaenv) mypc:~/Proj$ /home/me/.conda/condaenv/proj/bin/python3.7
Python 3.7.7 (default, Sep 24 2020, 16:28:06) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gym
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'gym'
>>> quit()

The environment variables are supposed to be the same as I run from the same location, on the same terminal, without altering it between the two calls. Since the first import works without any problem, the packages are well installed.

The system imports such as sys work fine in both version, but I had to export LD_LIBRARY_PATH before for the compiled version, which was not the case for the "normal" version. But keeping the LD_LIBRARY_PATH unchanged between the two calls don't change anything.

What happens that the second call can't find the matching package? What am I missing?

-- Edit 1 --

The compiled version shared objects dependencies are the following:

$ ldd python3.7
        linux-vdso.so.1 (0x00007ffed457a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63e53a6000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63e53a0000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f63e539b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f63e524c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63e505a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f63e577f000)

The original python are the following:

$ ldd python3.7.bak 
        linux-vdso.so.1 (0x00007ffc0d1c8000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f89ab6bc000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89ab4ca000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f89ab4c4000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f89ab4bf000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f89ab4b4000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f89ab365000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f89aba80000)

The versions differs a bit because they are not compiled with the same version of the compiler (I guess). I am however surprised that the new version doesn't require librt.so but the original version does. It could be because of the compilation flags though, which prevent the optimisation. I added --without-pymalloc --with-pydebug --with-valgrind during the configure step. But I don't think it should interfere with python normal behaviour with library.

Answer 1

The --without-pymalloc and --with-pydebug flags to ./configure are likely causing the error. Replacing the python3.7 binary in miniconda with one built with ./configure --without-pymalloc or ./configure --with-pydebug prevents the import of compiled libraries, like math and numpy. Compiling without any flags (i.e., ./configure && make) does not cause these errors. Compiling with ./configure --with-valgrind also does not raise errors when importing compiled libraries.

---

Older answer:

Can you please try with the configure flags used by conda-forge? Using these flags, I was able to replace the python3.7 binary in a conda environment, and imports continued to work. If I used the OP's flags (i.e., --without-pymalloc --with-pydebug --with-valgrind), then importing compiled libraries did not work (e.g., math, numpy).

curl -L https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tar.xz | tar xJ
cd Python-3.7.7
./configure \
    --prefix=/tmp/python3.7 \
    --enable-ipv6 \
    --with-ensurepip=no \
    --with-computed-gotos \
    --with-system-ffi \
    --enable-loadable-sqlite-extensions \
    --with-lto \
    --enable-optimizations \
    --with-valgrind
make -j

The solution involves one of these flags -- perhaps --with-lto or --enable-optimizations.

Answer 2

Where did you compiled the new Python? It could be that missing lib librt.so.1 is the culprit for the error message. Is there any chance to install the librt package (depends on your Linux flavor) and recompile Python.