Reputation: 41
I recently successfully embedded a python distribution with an application in Mac OS X using a homebrew installed python3.7 as per the methodology outlined in Joao Ventura's very useful two part series, provided here for reference (http://joaoventura.net/blog/2016/embeddable-python-osx/) and (http://joaoventura.net/blog/2016/embeddable-python-osx-from-src/).
The only remaining issue for me was to reduce the size of the python distribution size in the application by zip compressing the whole standard library minus lib-dynload
, config-3.7m-darwin
and site-packages
.
My directory structures is as follows:
- python3.7/
- include/
- lib/
- python3.7/
- libpython3.7.dylib
- python3.7 <executable>
The basic initial step is to move lib-dynload
and config-3.7m-darwin
from lib/python3.7
, so that I can compress the sodlib source files into lib/python37.zip
and then move lib-dynload
and config-3.7m-darwin
back into now empty lib/python3.7
to end up with the desired structure:
- python3.7/
- include/
- lib/
- python3.7/
- lib-dynload/
- config-3.7m-darwin
- python37.zip
- libpython3.7.dylib
- python3.7 <executable>
To test whether it worked or not, I would check sys.path
from the executable and try to import a module and check its __file__
attribute to see if it came from the zip archive.
On this basis, I would cd
into lib/python3.7
and try the following:
Select all files and folders and zip using OS X's Finder's compress
to generate python37.zip
Using the python zipfile
module:
python -m zipfile -c python37.zip lib/python3.7/*
zip
method from How can you bundle all your python code into a single zip file?cd lib/python3.7
zip -r9 ../python37.zip *
In all cases, I got it to work by setting PYTHONPATH to the zipped library, as in:
PYTHONPATH=lib/python37.zip ./python3.7`
Doing, I was able to successfully import from the zip archive and verify that the modules came from the zip archive. But without setting PYTHONPATH, it did not work.
Hence, I would very much appreciate some help to establish the correct and most straightforward way to zip the standard library such that it would be recognized automatically from sys.path (without any extra steps such as specifying the PYTHONPATH environment value which may not be possible on a user's machine).
Thanks in advance for any help provided.
S
Upvotes: 1
Views: 1474
Reputation: 41
Finally figured it out through a long process of elimination.
The only module you have to keep in site packages
or its parent folder is os.py
.
Here's a bash script for the whole process which is tested on macOS and can be easily modified for Linux as well.
It will download a given version of python as a source distribution from python.org and build it and install it locally in the same folder and then correctly zip the python standard library and test that it works.
Just run this script in any empty folder to test it.
#!/usr/bin/env bash
# build_zpython.sh
# builds minimized python with zipped `site-packages`
# NOTE: need os.py to remain in site-packages or just above it or it will fail
# ----------------------------------------------------------------------------
# VARS
NAME=zpython
VERSION=3.11.7
MAC_DEP_TARGET=10.13
# ----------------------------------------------------------------------------
PWD=$(pwd)
PREFIX=${PWD}/${NAME}
VER=${VERSION%.*}
VERN="${VER//./}"
LIB=${PREFIX}/lib/python${VER}
URL=https://www.python.org/ftp/python/${VERSION}/Python-${VERSION}.tar.xz
get_python() {
wget $URL
}
remove() {
echo "removing $1"
rm -rf $1
}
rm_lib() {
echo "removing $1"
rm -rf ${LIB}/$1
}
clean() {
echo "removing __pycache__ .pyc/o from $1"
find $1 | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf
}
clean_tests() {
echo "removing 'test' dirs from $1"
find $1 | grep -E "(tests|test)" | xargs rm -rf
}
clean_site_packages() {
echo "removing everything in $LIB/site-packages"
rm -rf $LIB/site-packages/*
}
rm_ext() {
echo "removing $LIB/lib-dynload/$1.cpython-${VER}-darwin.so"
rm -rf $LIB/lib-dynload/$1.cpython-*.so
}
rm_bin() {
echo "removing $PREFIX/bin/$1"
rm -rf $PREFIX/bin/$1
}
# ----------------------------------------------------------------------------
# main
main() {
if ! test -f Python-${VERSION}.tar.xz; then
wget $URL
fi
if test -d Python-${VERSION}; then
rm -rf Python-${VERSION}
fi
tar xvf Python-${VERSION}.tar.xz
cd Python-${VERSION}
./configure MACOSX_DEPLOYMENT_TARGET=${MAC_DEP_TARGET} \
--prefix=$PREFIX \
--enable-shared \
--disable-test-modules \
--with-ensurepip=no \
--without-static-libpython
make install
clean $PREFIX
clean_tests $LIB
clean_site_packages
remove ${LIB}/site-packages
# remove what you want here...
rm_lib config-${VERSION}-darwin
rm_lib idlelib
rm_lib lib2to3
rm_lib tkinter
rm_lib turtledemo
rm_lib turtle.py
rm_lib ensurepip
rm_lib venv
remove $LIB/distutils/command/*.exe
remove $LIB/lib-dynload/_xx*.so
remove $LIB/lib-dynload/xx*.so
remove $PREFIX/lib/pkgconfig
remove $PREFIX/share
# remove what you want here...
rm_ext _tkinter
rm_bin 2to3-${VER}
rm_bin idle${VER}
rm_bin idle3
rm_bin 2to3-${VER}
rm_bin 2to3
mv $LIB/lib-dynload $PREFIX
cp $LIB/os.py $PREFIX
clean $PREFIX
echo "zip site-packages"
python3 -m zipfile -c $PREFIX/lib/python${VERN}.zip $LIB/*
remove $LIB
mkdir -p $LIB
mv $PREFIX/lib-dynload $LIB
mv $PREFIX/os.py $LIB
mkdir $LIB/site-packages
echo "cleanup"
rm -rf Python-${VERSION}
}
test_zpython() {
./python/bin/python3 -c "import string; print(string.__file__)"
./python/bin/python3 -c "import string; assert string.digits=='0123456789'"
}
# run it
time main
# test it
test_zpython
Post feedback if you encounter any issues.
Upvotes: 1