CODEWITHSUNDEEP
pythonheroku
Alex Nikitin
Alex Nikitin

Reputation: 931

Heroku: Compiled Slug Size is too large Python

I trying to deploy my app to heroku

I have following deploying error

    transformers 4.8.2 requires huggingface-hub==0.0.12, but you'll have huggingface-hub 0.0.14 which is incompatible.
       Successfully installed MarkupSafe-2.0.1 Send2Trash-1.7.1 altair-4.1.0 argon2-cffi-20.1.0 astor-0.8.1 async-generator-1.10 attrs-21.2.0 backcall-0.2.0 base58-2.1.0 beautifulsoup4-4.9.3 bleach-3.3.1 blinker-1.4 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.3 click-7.1.2 cloudscraper-1.2.58 debugpy-1.3.0 decorator-5.0.9 defusedxml-0.7.1 entrypoints-0.3 filelock-3.0.12 gitdb-4.0.7 gitpython-3.1.18 huggingface-hub-0.0.14 idna-3.2 ipykernel-6.0.3 ipython-7.25.0 ipython-genutils-0.2.0 ipywidgets-7.6.3 jedi-0.18.0 jinja2-3.0.1 joblib-1.0.1 jsonschema-3.2.0 jupyter-client-6.1.12 jupyter-core-4.7.1 jupyterlab-pygments-0.1.2 jupyterlab-widgets-1.0.0 matplotlib-inline-0.1.2 mistune-0.8.4 nbclient-0.5.3 nbconvert-6.1.0 nbformat-5.1.3 nest-asyncio-1.5.1 nltk-3.6.2 notebook-6.4.0 numpy-1.21.0 packaging-21.0 pandas-1.2.5 pandocfilters-1.4.3 parso-0.8.2 pexpect-4.8.0 pickleshare-0.7.5 pillow-8.3.1 prometheus-client-0.11.0 prompt-toolkit-3.0.19 protobuf-3.17.3 ptyprocess-0.7.0 pycparser-2.20 pydeck-0.6.2 pygments-2.9.0 pyparsing-2.4.7 pyrsistent-0.18.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 pyzmq-22.1.0 regex-2021.7.6 requests-2.26.0 requests-toolbelt-0.9.1 sacremoses-0.0.45 scikit-learn-0.24.2 scipy-1.7.0 sentence-transformers-2.0.0 sentencepiece-0.1.96 six-1.16.0 smmap-4.0.0 soupsieve-2.2.1 streamlit-0.84.1 terminado-0.10.1 testpath-0.5.0 threadpoolctl-2.2.0 tokenizers-0.10.3 toml-0.10.2 toolz-0.11.1 torch-1.9.0 torchvision-0.10.0 tornado-6.1 tqdm-4.61.2 traitlets-5.0.5 transformers-4.8.2 typing-extensions-3.10.0.0 tzlocal-2.1 urllib3-1.26.6 validators-0.18.2 watchdog-2.1.3 wcwidth-0.2.5 webencodings-0.5.1 widgetsnbextension-3.5.1
-----> Downloading NLTK corpora…
 !     'nltk.txt' not found, not downloading any corpora
 !     Learn more: https://devcenter.heroku.com/articles/python-nltk
-----> Discovering process types
       Procfile declares types -> web
-----> Compressing...
 !     Compiled slug size: 1014.9M is too large (max is 500M).
 !     See: http://devcenter.heroku.com/articles/slug-size
 !     Push failed

I cannot figure out which library or dependency is too large that heroku cannot deploy it

Upvotes: 1

Views: 4645

Answers (1)

P0intMaN
P0intMaN

Reputation: 134

The maximum allowed slug size is 500MB. Slugs are an important aspect for heroku. When you git push to Heroku, your code is received by the slug compiler which transforms your repository into a slug.

First of all, lets determine what all files are taking up a considerate amount of space in your slug. To do that, fire up your heroku cli and enter / access your dyno by typing the following:

heroku run bash -a <appname>

Then, sort all the files present in the dyno by doing this:

du -ha --max-depth 1 /app | sort -hr

This would give you an idea about what files are taking up how much of a space.

Now, we can proceed to reduce the slug size. There are couple of ways through which you can reduce the slug size (assuming that you are using Heroku cli to deploy your application:

1. Using third party cloud storage: One of the popular ways in which you can reduce the slug size. Heroku supports Amazon S3 and Cloudinary among others. I prefer Amazon S3. Here is the documentation on how to setup S3 for Heroku. Move your large files and host them there. This would significantly reduce your slug size.

2. Using .slugignore : I can see that you are deploying some sort of ML application on heroku. Well, ML applications tend to be pretty heavy in size and frequently they cause problems during the deployment stage. Here is where .slugignore becomes very helpful. You can tell slug compiler to ignore some unnecessary files for slug compiler like images, test dataset, medias among many. More on how to use .slugignore can be found here in the official documentation

Upvotes: 5

Related Questions