Clean Uninstallation of Python Packages after Ubuntu Upgrade

Question

I recently upgraded a machine from Ubuntu Server 16.04 LTS to 18.04 LTS using command line. As a result, the built-in Python installation is upgraded from Python 3.5 to 3.6. And it seems that all the Python packages previously installed using pip3 are no longer accessible. I plan to clean up all those packages and use conda for Python package management. My question is, what is the best practice for a clean uninstallation of those inaccessible packages in this case?

The old packages installed via pip3 were primarily located under /usr/local/lib/python3.5/ and ~/.local/lib/python3.5/. But there could be other leftover files, e.g., under /usr/local/bin/. I would like to remove all of related files that came with pip3 install.

Answer 1

I have recently upgraded from Debian 11 to Debian 12 and forgot to remove the old Python (3.9) packages before the upgrade. After the upgrade, I was stuck with Python 3.11 trying to use user packages previously installed with Python 3.9, and I also had a bunch of executables in ~/.local/bin installed by the old Pip. The pip3 command itself was pointing to ~/.local/bin. Not ideal!

I came across this Q/A and I improved and adapted the script from GZ0. Hopefully it will help someone in the future. Here it is:

#!/bin/bash
#
# Marco Bonelli - 2024-04-28
#
# Purge user packages belonging to an old python version that is no longer
# installed after a dist-upgrade. Deleting the entire ~/.local/lib/pythonX.Y
# directory may seem enough, but it really is not as a lot of packages may also
# install executables in ~/.local/bin (and maybe even other additional stuff).
#
# Adapted from: https://stackoverflow.com/a/56960145/3889449
#

# Set this to 'NO' to actually uninstall packages
DRY_RUN='YES'

# Old python version to destroy
VICTIM_VERSION='3.9'

# We will look for site-packages and dist-packages here
VICTIM_LIB_PATH="$HOME/.local/lib/python$VICTIM_VERSION"

log() {
    echo -ne '\033[94m'
    echo -n "$@"
    echo -e '\033[0m'
}

uninstall_kind() {
    export PYTHONPATH="$VICTIM_LIB_PATH/$1"

    # We ideally whould use the pip of the corresponding python version we are
    # uninstalling packages for. Just being cautious here, it may work anyway.
    # You can remove this check if you really want.
    if ! [[ "$(which pip3)" != *"$VICTIM_LIB_PATH"* ]]; then
        echo "WARNING: pip3 does not point into $VICTIM_LIB_PATH" >&2
        echo "WARNING: not sure if this could break stuff... aborting!" >&2
        exit 1
    fi

    if [ "$DRY_RUN" = 'NO' ]; then
        # --break-system-packages is needed for distros like Debian 12 that set
        # their Python installation as "externally managed" to avoid conflicts
        # between python packages installed through apt and pip. We are not
        # deleting system packages anyway as this script is not run as root, so
        # it's safe to use it.
        local uninstall_cmd="pip3 uninstall -y --break-system-packages"
    else
        local uninstall_cmd="log [dry run] pip3 uninstall -y --break-system-packages"
    fi

    log "Uninstalling packages in $PYTHONPATH"

    # List all packages
    pip3 freeze --all --local | cut --delimiter="=" -f 1 | while read pkg ; do
        # Don't shoot ourselves in the foot by uninstalling pip while using it
        if [ "$pkg" = "pip" ]; then
            log 'Skipping pip (delete it only at the end)'
            continue
        fi

        # Ask pip to resolve the package
        local loc="$(pip3 show $pkg | awk '/Location:/ { print $2 }')"

        # If the identified location is within $VICTIM_LIB_PATH then uninstall,
        # otherwise skip. This is because we can still very well have listed
        # packages under /usr/local/lib and we don't want to remove those (it
        # would also fail since we don't have privileges).
        if [[ "$loc" == *"$VICTIM_LIB_PATH"* ]]; then
            log "Uninstalling $pkg ($loc)"
            $uninstall_cmd $pkg
        else
            log "Skipping $pkg (resolves to $loc)"
        fi
    done
}

# Abort if run as root, we only want to deal with user packages
if [ "$EUID" -eq 0 ]; then
    echo "This script should not be run as root!" >&2
    exit 1
fi

for kind in site-packages dist-packages; do
    uninstall_kind $kind
done

# Do the last steps manually
echo '----------------------------------------------------------------------'
echo 'Done! You can now safely delete the following:'
echo '- The pip and pip3 commands if they point to ~/.local/bin (see `which pip` and `which pip3`)'
echo "- The directory ~/.local/lib/python$VICTIM_VERSION"

Answer 2

I ended up writing a bash script to call pip3 uninstall on each previously installed package iteratively.

#!/bin/bash

pypath_cmd="PYTHONPATH=$HOME/.local/lib/python3.5/site-packages"
export $pypath_cmd
echo "Uninstalling editable packages in $PYTHONPATH"
rm -f $PYTHONPATH/*.egg-link
rm -f $PYTHONPATH/easy-install.pth

pip3 freeze --all --local | cut --delimiter="=" -f 1 | while read pkg ; do
    echo $pkg: $(pip3 show $pkg | grep "Location:")
    pip3 uninstall -y $pkg
done

pypath_cmd="PYTHONPATH=/usr/local/lib/python3.5/dist-packages"
export $pypath_cmd
echo "Uninstalling editable packages in $PYTHONPATH"
sudo rm -f $PYTHONPATH/*.egg-link
sudo rm -f $PYTHONPATH/easy-install.pth

pip3 freeze --all --local | cut --delimiter="=" -f 1 | while read pkg ; do
    echo $pkg: $(pip3 show $pkg | grep "Location:")
    sudo $pypath_cmd pip3 uninstall -y --no-cache-dir $pkg
done

Answer 3

sudo pip install installs pip packages to/usr/local/lib/<python_version>/dist-packages, and apt packages to /usr/lib/<python_version>/dist-packages. Check these directories and remove the unwanted packages.