CODEWITHSUNDEEP
pythonpip
Alex Dalyac
Alex Dalyac

Reputation: 7044

How to determine the size of a non-installed python package?

I don't have much space left, and I am tempted to install python libraries.

I would like to know how big a python library is before installing is (just to know how much extra space to request if I don't have enough).

Is there a way to do that, e.g. using pip?

Upvotes: 7

Views: 7127

Answers (3)

rubo77
rubo77

Reputation: 20865

If you have a requirements.txt file use this script:

#!/bin/bash
# by rubo77: https://stackoverflow.com/a/68278901/1069083

mkdir -p /tmp/pip-size
for i in $(pip-sync -n requirements.txt|tail -n +2); do
  echo -ne "$i:\t"
  wget --quiet -O /tmp/pip-size/$i https://pypi.org/pypi/$i/json
  cat /tmp/pip-size/$i | jq 'last(.releases[])'[].size | paste -sd+ | bc > /tmp/pip-size/$i.size
  cat /tmp/pip-size/$i.size | awk '{$1=$1/1024/1024; print $1,"MB";}'
done
echo -e "--------\ntotal:"
cat /tmp/pip-size/*.size  | paste -sd+ | bc | awk '{$1=$1/1024/1024; print $1,"MB";}';

output example:

ninja:  1.4366 MB
numpy:  120.735 MB
opencv-python:  724.788 MB
pillow: 137.695 MB
scikit-image:   7.46707 MB
torch:  4772.73 MB
torchvision:    153.842 MB
--------
total:
5918.69 MB

Upvotes: 2

aqm
aqm

Reputation: 3034

This is my script i'm currently using

(based off the excellent package walking code from How to see pip package sizes installed?)

Steps :

  1. create a python script to check all currently installed pip packages
  2. create a shell script to create a brand new python environment and install package to test
  3. run shell script
  4. profit :)

Step 1

create a python script called tool-size.py

#!/usr/bin/env python

import os
import pkg_resources

def calc_container(path):
    total_size = 0
    for dirpath, dirnames, filenames in os.walk(path):
        for f in filenames:
            fp = os.path.join(dirpath, f)
            total_size += os.path.getsize(fp)
    return total_size

def calc_installed_sizes():
    dists = [d for d in pkg_resources.working_set]

    total_size = 0
    print (f"Size of Dependencies")
    print("-"*40)
    for dist in dists:
        # ignore pre-installed pip and setuptools
        if dist.project_name in ["pip", "setuptools"]:
            continue
        try:
            path = os.path.join(dist.location, dist.project_name)
            size = calc_container(path)
            total_size += size
            if size/1000 > 1.0:
                print (f"{dist}: {size/1000} KB")
                print("-"*40)
        except OSError:
            '{} no longer exists'.format(dist.project_name)

    print (f"Total Size (including dependencies): {total_size/1000} KB")

if __name__ == "__main__":
    calc_installed_sizes()

Step 2

create a bash script called tool-size.sh

#!/usr/bin/env bash

# uncomment to to debug
# set -x

rm -rf ~/.virtualenvs/tool-size-tester
python -m venv ~/.virtualenvs/tool-size-tester
source ~/.virtualenvs/tool-size-tester/Scripts/activate
pip install -q $1
python tool-size.py
deactivate

Step 3

run script with package you want to get the size of

tool-size.sh xxx

say for truffleHog3

$ ./tool-size.sh truffleHog3

Size of Dependencies
----------------------------------------
truffleHog3 2.0.6: 56.46 KB
----------------------------------------
smmap 4.0.0: 108.808 KB
----------------------------------------
MarkupSafe 2.0.1: 40.911 KB
----------------------------------------
Jinja2 3.0.1: 917.551 KB
----------------------------------------
gitdb 4.0.7: 320.08 KB
----------------------------------------
Total Size (including dependencies): 1443.81 KB

Upvotes: 1

Simeon Visser
Simeon Visser

Reputation: 122436

This is not possible using pip as far as I know. But if the package is hosted on PyPI you can append /json to the URL to view more details, including file size.

For example, visit https://pypi.python.org/pypi/pip/json and have a look at the entries in the releases key. The size of each release tells you how big the download would be.

Note that this tells you the download size, not the size after installing (as a package may request additional dependencies for installation, after decompressing the archives, and so on). So your question may be difficult to answer, prior to installation.

Upvotes: 11

Related Questions