CODEWITHSUNDEEP
pythonlinuxdockersslopenssl
Jake Stevens-Haas
Jake Stevens-Haas

Reputation: 1696

Building Python and OpenSSL from source, but ssl module fails

I'm trying to build Python and OpenSSL from source in a container. Both seem to build correctly, but Python does not successfully create the _ssl module.

I've found a few guides online that say to un-comment and lines from Python-3.X.X/Modules/Setup and add the --openssldir=/usr/local/ssl flag to the ./configure step for OpenSSL. I do these in my dockerfile. This has had the effect that, during the ./configure output for Python, I see the following line. 

checking for X509_VERIFY_PARAM_set1_host in libssl... yes

Yet I receive the following errors:

[91m*** WARNING: renaming "_ssl" since importing it failed: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by build/lib.linux-x86_64-3.8/_ssl.cpython-38-x86_64-linux-gnu.so)
[0m[91m*** WARNING: renaming "_hashlib" since importing it failed: /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1: version `OPENSSL_1_1_1' not found (required by build/lib.linux-x86_64-3.8/_hashlib.cpython-38-x86_64-linux-gnu.so)
[0m
Python build finished successfully!

...

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl                                     


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

If ./configure finds X509..., why am I still getting the hashlib and ssl errors?

The full Dockerfile, FWIW:

FROM jenkins/jenkins:lts
USER root
RUN apt-get update && apt-get install -y apt-utils gcc make zlib1g-dev \
    build-essential libffi-dev checkinstall libsqlite3-dev 
RUN wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz && \
    tar xzf openssl-1.1.1d.tar.gz && \
    cd openssl-1.1.1d && \
    ./config -Wl,--enable-new-dtags,-rpath,'$(LIBRPATH)' --prefix=/usr/local/ssl --openssldir=/usr/local/ssl && \
    make && \
    make test && \
    make install
RUN wget -q https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz && \
    tar -xzf Python-3.8.2.tgz && \
    cd Python-3.8.2 && \
    ./configure && \
    make && \
    make install
USER jenkins

Upvotes: 7

Views: 28759

Answers (4)

kris
kris

Reputation: 23591

The following worked for me on Amazon's EC2, with the default CentOS 7 system.

First, the openssl libraries on CentOS 7 are too old (Python 3.9+ wants openssl 1.1.1+ and the version available is 1.0.x). Install the newer ones:

sudo yum install openssl11-devel

Note: since writing this answer, Amazon has end-of-life'd the openssl11-devel package and updated openssl-devel to 3.0.8. 3.0.8 is more than enough for Python, so now you can just do yum install openssl-devel.

Unfortunately, CentOS doesn't actually put the SSL libraries anywhere that Python can find them. With some trial and error, I found that this makes ./configure happy:

export OPENSSL_LIBS=/usr/lib64/libssl.so
./configure \
  --with-openssl=/usr \
  --with-openssl-rpath=/usr/lib64 \
  --enable-optimizations

Explanation:

  • Python doesn't look in lib64 by default, which is where openssl11 is installed.
  • --with-openssl takes a path that ./configure appends include/openssl/ssl.h to, so make sure that is there for your system!

When you run ./configure, you're looking for a line near the end of the output like:

checking for stdlib extension module _ssl... yes

If you see missing instead of yes, search config.log for openssl, which should give you some guidance about where it's screwing up.

Hopefully this saves someone else the many hours I spent figuring this out.

Upvotes: 15

Marc Maxmeister
Marc Maxmeister

Reputation: 4699

I had the same problem and after 3+ hours of searching THIS is what actually worked:

Error: "ssl module is not available" when installing package with pip3
———————————————————-
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/pip/

The fix:
———————————————————-
sudo yum install openssl11 openssl11-devel
mkdir /usr/local/openssl11
cd /usr/local/openssl11
ln -s /usr/lib64/openssl11 lib
ln -s /usr/include/openssl11 include

The above code creates an alternate path for the latest openssl11 and symlinks it in a place that matches the folder structure that python expects.

Then proceed with the original steps in the guide (https://linuxstans.com/how-to-install-python-centos/)

and ./configure like this within your python install folder:

./configure -–enable-optimizations -–with-openssl=/usr/local/openssl11
make altinstall

the altinstall is important so that you don't overwrite the system default python. You'll have to invoke it as python3.10 from command line thereafter.

I tested this on CentOS 7 and Python-3.10.8.

ref: https://linuxstans.com/how-to-install-python-centos/?unapproved=4338&moderation-hash=9903c8ebd6634e5bbbad96716e283f8b#comment-4338

Upvotes: 7

T.H.
T.H.

Reputation: 876

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl                                     

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

It seems like installation issue when building openssl from source. For build failure on _ssl module, try extra options like --with-openssl, CFLAGS and LDFLAGS when configuring Python using the script ./configure, e.g. 

./configure  --with-openssl=/PATH/TO/YOUR/OPENSSL_INSTALL_FOLDER/ \
    --enable-optimizations \
    --with-ssl-default-suites=openssl \
    CFLAGS="-I/PATH/TO/YOUR/OPENSSL_INSTALL_FODLER/include" \
    LDFLAGS="-L/PATH/TO/YOUR/OPENSSL_INSTALL_FODLER/"

Also try this command openssl version, if it reports error like this :

/usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found

that means there is linking problem on your openssl library, I'm not sure if you're on Linux or other system, but for Linux system, you can manually modify the links to openssl library to fix the problem as described in my answer at here.

Reference

Building Python 3.7.1 - SSL module failed

Python 3.7.0 wont compile with SSL Support 1.1.0

Upvotes: 7

Max Luchterhand
Max Luchterhand

Reputation: 498

I reckon that Jenkins Image comes with some openssl version installed that is not 1.1.1, hence you find X509... in libssl but cant build.

Regarding said config option, you can spin up the container with bash as CMD, copy the config from within the container to the machine where the Image lies, edit ist and bake your version of the config into the Image.

Upvotes: 1

Related Questions