shakfu
shakfu

Reputation: 41

How to correctly zip the whole python standard library?

I recently successfully embedded a python distribution with an application in Mac OS X using a homebrew installed python3.7 as per the methodology outlined in Joao Ventura's very useful two part series, provided here for reference (http://joaoventura.net/blog/2016/embeddable-python-osx/) and (http://joaoventura.net/blog/2016/embeddable-python-osx-from-src/).

The only remaining issue for me was to reduce the size of the python distribution size in the application by zip compressing the whole standard library minus lib-dynload, config-3.7m-darwin and site-packages.

My directory structures is as follows:

- python3.7/
  - include/
  - lib/
    - python3.7/
  - libpython3.7.dylib
  - python3.7 <executable>

The basic initial step is to move lib-dynload and config-3.7m-darwin from lib/python3.7, so that I can compress the sodlib source files into lib/python37.zip and then move lib-dynload and config-3.7m-darwin back into now empty lib/python3.7 to end up with the desired structure:

- python3.7/
  - include/
  - lib/
    - python3.7/
      - lib-dynload/
      - config-3.7m-darwin
    - python37.zip
  - libpython3.7.dylib
  - python3.7 <executable>

To test whether it worked or not, I would check sys.path from the executable and try to import a module and check its __file__ attribute to see if it came from the zip archive.

On this basis, I would cd into lib/python3.7 and try the following:

  1. Select all files and folders and zip using OS X's Finder's compress to generate python37.zip

  2. Using the python zipfile module:

python -m zipfile -c python37.zip lib/python3.7/*

  1. Using the zip method from How can you bundle all your python code into a single zip file?
cd lib/python3.7
zip -r9 ../python37.zip *

In all cases, I got it to work by setting PYTHONPATH to the zipped library, as in:

PYTHONPATH=lib/python37.zip ./python3.7`

Doing, I was able to successfully import from the zip archive and verify that the modules came from the zip archive. But without setting PYTHONPATH, it did not work.

Hence, I would very much appreciate some help to establish the correct and most straightforward way to zip the standard library such that it would be recognized automatically from sys.path (without any extra steps such as specifying the PYTHONPATH environment value which may not be possible on a user's machine).

Thanks in advance for any help provided.

S

Upvotes: 1

Views: 1474

Answers (1)

shakfu
shakfu

Reputation: 41

Finally figured it out through a long process of elimination.

The only module you have to keep in site packages or its parent folder is os.py.

Here's a bash script for the whole process which is tested on macOS and can be easily modified for Linux as well.

It will download a given version of python as a source distribution from python.org and build it and install it locally in the same folder and then correctly zip the python standard library and test that it works.

Just run this script in any empty folder to test it.

#!/usr/bin/env bash

# build_zpython.sh
# builds minimized python with zipped `site-packages`
# NOTE: need os.py to remain in site-packages or just above it or it will fail

# ----------------------------------------------------------------------------
# VARS

NAME=zpython
VERSION=3.11.7
MAC_DEP_TARGET=10.13

# ----------------------------------------------------------------------------

PWD=$(pwd)
PREFIX=${PWD}/${NAME}
VER=${VERSION%.*}
VERN="${VER//./}"
LIB=${PREFIX}/lib/python${VER}
URL=https://www.python.org/ftp/python/${VERSION}/Python-${VERSION}.tar.xz


get_python() {
    wget $URL
}


remove() {
    echo "removing $1"
    rm -rf $1
}

rm_lib() {
    echo "removing $1"
    rm -rf ${LIB}/$1
}


clean() {
    echo "removing __pycache__ .pyc/o from $1"
    find $1 | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf
}

clean_tests() {
    echo "removing 'test' dirs from $1"
    find $1 | grep -E "(tests|test)" | xargs rm -rf
}

clean_site_packages() {
    echo "removing everything in $LIB/site-packages"
    rm -rf $LIB/site-packages/*
}

rm_ext() {
    echo "removing $LIB/lib-dynload/$1.cpython-${VER}-darwin.so"
    rm -rf $LIB/lib-dynload/$1.cpython-*.so
}

rm_bin() {
    echo "removing $PREFIX/bin/$1"
    rm -rf $PREFIX/bin/$1
}

# ----------------------------------------------------------------------------
# main

main() {
    if ! test -f Python-${VERSION}.tar.xz; then
        wget $URL
    fi
    if test -d Python-${VERSION}; then
        rm -rf Python-${VERSION}
    fi
    tar xvf Python-${VERSION}.tar.xz
    cd Python-${VERSION}

    ./configure MACOSX_DEPLOYMENT_TARGET=${MAC_DEP_TARGET} \
        --prefix=$PREFIX \
        --enable-shared \
        --disable-test-modules \
        --with-ensurepip=no \
        --without-static-libpython

    make install

    clean $PREFIX
    clean_tests $LIB
    clean_site_packages
    remove ${LIB}/site-packages

    # remove what you want here...
    rm_lib config-${VERSION}-darwin
    rm_lib idlelib
    rm_lib lib2to3
    rm_lib tkinter
    rm_lib turtledemo
    rm_lib turtle.py
    rm_lib ensurepip
    rm_lib venv

    remove $LIB/distutils/command/*.exe
    remove $LIB/lib-dynload/_xx*.so
    remove $LIB/lib-dynload/xx*.so
    remove $PREFIX/lib/pkgconfig
    remove $PREFIX/share

    # remove what you want here...
    rm_ext _tkinter


    rm_bin 2to3-${VER}
    rm_bin idle${VER}
    rm_bin idle3
    rm_bin 2to3-${VER}
    rm_bin 2to3


    mv $LIB/lib-dynload $PREFIX
    cp $LIB/os.py $PREFIX
    clean $PREFIX

    echo "zip site-packages"
    python3 -m zipfile -c $PREFIX/lib/python${VERN}.zip $LIB/*
    remove $LIB
    mkdir -p $LIB
    mv $PREFIX/lib-dynload $LIB
    mv $PREFIX/os.py $LIB
    mkdir $LIB/site-packages

    echo "cleanup"
    rm -rf Python-${VERSION}
}

test_zpython() {
    ./python/bin/python3 -c "import string; print(string.__file__)"
    ./python/bin/python3 -c "import string; assert string.digits=='0123456789'"
}


# run it
time main

# test it
test_zpython

Post feedback if you encounter any issues.

Upvotes: 1

Related Questions