CODEWITHSUNDEEP
pythonoperating-systempipconda
legel
legel

Reputation: 2673

Best way to distribute big, complex Python projects so that they fully install for other users, regardless of operating system?

How to ensure that distributing Python packages with big dependencies, like NumPy and SciPy, automatically get properly installed on users' computers, regardless of their OS?

The problem is that many such packages are installed with more or less ease through the main channel of pip, including those mentioned above...

Upvotes: 4

Views: 578

Answers (2)

Zoro_77
Zoro_77

Reputation: 415

The problem with legel's suggestion is that you still end up relying on user's environment to deploy your application.

In the past I have had similar install scripts which would do all the dirty work to deploy my application, but it still relies on network connectivity to download packages and dependencies. I have gotten past that by creating a docker container. I would build a container with my application and all the dependencies, create a new docker image and provide the docker image to customers. The only dependency then is to have docker on the system, and the way to run my app would be to use docker run.

There is another way to package all dependencies is to use http://www.pyinstaller.org/ which creates a standalone application packaging all dependent modules. Disclaimer though: I have not used this on any large applications yet, have just started testing it, and so far works for smaller programs.

Hope this helps!

Upvotes: 0

legel
legel

Reputation: 2673

I don't claim this is perfect, but below is what I've found in practice to be really strong: an install.sh script, derived here originally for an artificial intelligence library called words2map. The goal was to make sure that users can simply run ./install.sh and then all the needed pip install this and pip install that succeeds in every case (even for brand new servers), regardless of the operating system you're on. (I really wish OS-agnostic installations was industry standard, at least across Mac OSX and every Linux distro...)

Feel free to skip straight to the code, but for those who demand explanation: This install.sh script relies heavily on the Conda package distribution manager, which works really nicely, given that it builds OS-specific wheels automatically. The installation code below begins by first checking the OS of the user, then installing the appropriate Conda distribution (if on Mac OSX and Linux, in this case), and then finally installing all of the remaining dependencies as needed through Conda. Also, note that if a Linux terminal is detected it first ensures the needed compilation libraries are available through e.g. sudo apt-get install python-dev (since often users are missing gcc or something), and finally this automatically adds Conda to the user's PATH for a bash and zshell, which seems to cover most terminals (but maybe not, feel free to add). A lot of bells and whistles, but in practice this delivers a really nice UX for helping users immediately get up and running with a new complex Python library you're distributing, as you could test for yourself by trying to immediately download and run words2map. Edits to this are strongly encouraged if they make sense / make this better / clearer for everyone. Okay, on to the show!

#!/bin/bash

download_miniconda() {
    echo "Downloading Miniconda for Python dependencies..."
    OS_BIT_TYPE="$(uname -m)"
    OS_ARCHITECTURE="$(uname -s)"
    if [ $OS_BIT_TYPE == "i686" ]; then
        OS_BIT_TYPE="x86"
    fi
    if [ $OS_ARCHITECTURE == "Darwin" ]; then
        OS_ARCHITECTURE="MacOSX"
    fi

    MINICONDA_INSTALL_FILE="Miniconda2-latest-$OS_ARCHITECTURE-$OS_BIT_TYPE.sh"
    MINICONDA_DOWNLOAD_URL="https://repo.continuum.io/miniconda/$MINICONDA_INSTALL_FILE"
    $(curl -O $MINICONDA_DOWNLOAD_URL)
    $(chmod +x $MINICONDA_INSTALL_FILE)
}

install_miniconda() {
    echo "Installing Miniconda..."
    echo "$(./$MINICONDA_INSTALL_FILE -b -p $HOME/miniconda)"
    echo "$(rm $MINICONDA_INSTALL_FILE)"    
}

confirm_miniconda_installed() {
    if hash conda 2>/dev/null; then
        echo "Miniconda installed!"
    else
        echo "Failed to install Miniconda. Please visit http://conda.pydata.org/docs/install/quick.html to install and then try rerunning this script, making sure that Miniconda is accessible in the PATH"
    fi
}

update_script_startup_file() {
    echo "if [[ \":\$PATH:\" != *\":\$HOME/miniconda/bin:\"* ]]; then" >> $STARTUP_FILE
    echo "  export PATH=\"\$PATH:\$HOME/miniconda/bin\"" >> $STARTUP_FILE
    echo "fi" >> $STARTUP_FILE
}

add_miniconda_to_path() {
    # temporary update to PATH for this script
    export PATH="$PATH:$HOME/miniconda/bin"

    # permanent update to PATH for user's convenience
    if [ -n "`$SHELL -c 'echo $BASH_VERSION'`" ]; then
        STARTUP_FILE="$HOME/.bashrc"
        update_script_startup_file
    elif [ -n "`$SHELL -c 'echo $ZSH_VERSION'`" ]; then
        STARTUP_FILE="$HOME/.zshrc"
        update_script_startup_file
    else
        echo "Couldn't automatically add Miniconda to the PATH of your preferred terminal. We suggest working from Bash or ZShell." 
    fi
}

install_conda_if_needed() {
    if hash conda 2>/dev/null; then
        echo "Miniconda installed!"
    else
        if ping -c 1 google.com >> /dev/null 2>&1; then
            download_miniconda
            install_miniconda
            add_miniconda_to_path
            confirm_miniconda_installed
        else
            echo "Looks like you're offline! Please address this and then try rerunning this script."
        fi
    fi
}

create_conda_environment() {
    if hash conda 2>/dev/null; then
        CONDA_ENVIRONMENTS="$(conda env list)"
        if [[ "$CONDA_ENVIRONMENTS" != *"words2map"* ]]; then
            conda create --name words2map --yes cython scikit-learn gensim seaborn
        fi
    fi
}

install_developer_libraries_as_needed() {
    OS_ARCHITECTURE="$(uname -s)"
    if [ $OS_ARCHITECTURE == "Linux" ]; then
        echo "$(python -mplatform | grep -qi Ubuntu && sudo apt-get update && sudo apt-get install python-dev || sudo yum update -y && sudo yum install python-devel -y && sudo yum groupinstall "Development Tools" -y)"
    fi
}

install_python_dependencies() {
    if hash conda 2>/dev/null; then
        echo 'Installing Python dependencies for words2map...'
        source activate words2map
        install_developer_libraries_as_needed
        pip install hdbscan pattern semidbm nltk unidecode
        source deactivate
    fi  
}

refresh_user_shell() {
    if [ -n "`$SHELL -c 'echo $BASH_VERSION'`" ]; then
        exec bash
    elif [ -n "`$SHELL -c 'echo $ZSH_VERSION'`" ]; then
        exec zsh
    fi
}

install_conda_if_needed
create_conda_environment
install_python_dependencies
refresh_user_shell

Upvotes: 4

Related Questions