How to resolve a conflict between Debian 12 and Automake regarding the directory where compiled and packaged Python modules are installed?

Question

First, some context. The issue is related to the migration of a package from Ubuntu 16.04 to Debian 12. In the meantime, it was successfully (almost out of the box) installed and tested on Debian 10 by importing the Xenial repositories. Packages in “PPA for SynCE”

I have a problem with some Python-related directory variables: pythondir and pyexecdir. I understand, these define the subdirectory of the Python install tree in which to install Python scripts and extension modules (shared libraries) should be installed.

By default, on Debian 10 (with Python 2.7), the path is $PYTHON_PREFIX/lib/pythonversion/dist-packages, where $PYTHON_PREFIX is /usr and name of directory is based on distutils.

On Debian 12, Python 3.11 have their own installation scheme, based on posix_prefix (force to use /usr over /usr/local Debian standar), with prevalence of sysconfig over disutils (deprecated since 3.10 and marked for removal in Python 3.12). In this context, Automake enforces the site-packages scheme, which causes me problems when building packages because devscripts expects to find libraries in dist-packages.

This is what the code looks like, very simplified:

if can_use_sysconfig:
    python3 -c "import sys; import sysconfig; sys.stdout.write(sysconfig.get_path('purelib', 'posix_prefix', vars={'base':'/usr'}))"
    (result: /usr/lib/python3.11/site-packages)
else:
    python3 -c "import sys; from distutils import sysconfig; sys.stdout.write(sysconfig.get_python_lib(0, 0, prefix='/usr'))" 2>/dev/null
    (result: /usr/lib/python3/dist-packages)

When I build the deb package, I get the following messages:

checking for GNU default /usr/bin/python3.11 prefix... ${prefix}
checking for GNU default /usr/bin/python3.11 exec_prefix... ${exec_prefix}
checking for /usr/bin/python3.11 script directory (pythondir)... ${PYTHON_PREFIX}/lib/python3.11/site-packages
checking for /usr/bin/python3.11 extension module directory (pyexecdir)... ${PYTHON_EXEC_PREFIX}/lib/python3.11/site-packages
...
libtool: install: /usr/bin/install -c .libs/pyrtfcomp.soT ~/librtfcomp-1.3/debian/tmp/usr/lib/python3.11/site-packages/pyrtfcomp.so
...
dh_python3 -ppython-rtfcomp
E: dh_python3 dh_python3:197: no package to act on (python3-foo or one with ${python3:Depends} in Depends)
...
Now running lintian librtfcomp_1.3-1debian1~bookworm1_amd64.changes ...
E: python-rtfcomp: python-package-missing-depends-on-python
W: python-rtfcomp: python-module-in-wrong-location usr/lib/python3.11/site-packages/pyrtfcomp.so -> usr/lib/python3/dist-packages/pyrtfcomp.so
Finished running lintian.

How should I modify Automake scripts to follow Python's posix_prefix scheme policy and override disutils but without conflicting with the path expected by dh python3: /usr/lib/python3/dist-packages?

Some thoughts after posting the question...

Description of this distribution and python version

$ lsb_release -d
Description:    Debian GNU/Linux 12 (bookworm)

$ python3 -V
Python 3.11.2

purelib and platlib paths, in Python 3 sysconfig module, point to dist-packages directory:

$ python3 -m sysconfig
Platform: "linux-x86_64"
Python version: "3.11"
Current installation scheme: "posix_local"

Paths:
        data = "/usr/local"
        include = "/usr/include/python3.11"
        platinclude = "/usr/include/python3.11"
        platlib = "/usr/local/lib/python3.11/dist-packages"
        platstdlib = "/usr/lib/python3.11"
        purelib = "/usr/local/lib/python3.11/dist-packages"
        scripts = "/usr/local/bin"
        stdlib = "/usr/lib/python3.11"

One severe consequence of this is that Automake is installing Python modules to a location which is not in sys.path, namely site-packages directory.

  AC_CACHE_CHECK([for $am_display_PYTHON script directory (pythondir)],
  [am_cv_python_pythondir],
  [if test "x$am_cv_python_prefix" = x; then
     am_py_prefix=$am__usable_prefix
   else
     am_py_prefix=$am_cv_python_prefix
   fi
   am_cv_python_pythondir=`$PYTHON -c "
$am_python_setup_sysconfig
if can_use_sysconfig:                               <<------------- Replacing distutils
  if hasattr(sysconfig, 'get_default_scheme'):
    scheme = sysconfig.get_default_scheme()         <<------------- Result:posix_local
  else:
    scheme = sysconfig._get_default_scheme()                
  if scheme == 'posix_local':                       <<------------- Yes!
    # Debian's default scheme installs to /usr/local/ but we want to find headers in /usr/
    scheme = 'posix_prefix'                         <<------------- Here the scheme changes
  sitedir = sysconfig.get_path('purelib', scheme, vars={'base':'$am_py_prefix'})
  // Output: /usr/lib/python3.11/site-packages      <<------------- sitedir assumes its value here
else:
  from distutils import sysconfig
  sitedir = sysconfig.get_python_lib(0, 0, prefix='$am_py_prefix')  <<------------- This branch would need to redirect warning messages 
  // The exit here would be /usr/lib/python3/dist-packages but...   <<------------- to /dev/null to finish without error.

sys.stdout.write(sitedir)"`

If I change to the scheme posix_prefix, sysconfig module point to site-packages directory

$ python3 -c "import sysconfig; print(sysconfig.get_path('purelib', 'posix_prefix'))"
/usr/lib/python3.11/site-packages

Here, it can be seen that automake uses purelib path defined by the sysconfig module of Python. However, since this path is not included in sys.path, and thus the installed package cannot be used.

$ python3 -c "import sys; print(sys.path)"
['', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']

Additionally if for some reason the sysconfig module cannot be used, automake scripts will fallback to using distutils.sysconfig like this:

$ python3 -c "from distutils import sysconfig; print(sysconfig.get_python_lib(0, 0, prefix='/usr'))"
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
/usr/lib/python3/dist-packages

This is correctly set to dist-packages. However...

1. sys.path contains /usr/lib/python3/dist-packages not /usr/local/lib/python3.11/dist-packages Hence the modules are not installed in the correct path unless a symlink python3->python3.11 is manually made.
2. Uncaptured messages of DeprecationWarning cause unrepairable error
3. Marked for removal in Python 3.12

Answer 1

Thanks to the following bug report: https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1408092

And email related: https://lists.ubuntu.com/archives/foundations-bugs/2017-June/323130.html

With this, I arrived at the conclusion that the change had to be made into the file /usr/share/aclocal-1.16/python.m4

According to the Python documentation https://peps.python.org/pep-0632/#migration-advice, instead of distutils.sysconfig I had to use sysconfig, but the options of this one did not give me the same output as distutils. My best option was to use sys.path, specifically the last item in the list:

$ python3 -c "import sys; print(sys.path)"

['', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']

Replacing the two lines:

sitedir = sysconfig.get_path('purelib', scheme, vars={'base':'$am_py_prefix'})

With:

sitedir = sys.path[[-1]]

Both changes correspond to the [am_cv_python_pythondir] and [am_cv_python_pyexecdir] sections of the code. I found the 3rd Autotools video very helpful where they talk about escaping the [ ]

https://youtu.be/e-uYBb554LU?feature=shared&t=337

I replaced the line related to the pythondir variable and left the one related to pyexecdir so I could see the difference, then ran autoreconf -i and debuild..

checking for /usr/bin/python3.11 script directory (pythondir)... 
${PYTHON_PREFIX}/lib/python3/dist-packages
checking for /usr/bin/python3.11 extension module directory (pyexecdir)... 
${PYTHON_EXEC_PREFIX}/lib/python3.11/site-packages

Eureka!

Another resource: https://www.gnu.org/software/automake/manual/html_node/Python.html