CODEWITHSUNDEEP
pythonpipaws-lambda
MLu
MLu

Reputation: 1380

Prevent pip from installing some dependencies

We're developing an AWS Lambda function for Alexa skill in Python and using pip to install the ask-sdk package to our dist/ directory:

pip install -t dist/ ask-sdk

The trouble is with the -t dist/ because pip wants to have all the dependencies there, even if they are installed system-wide.

Now, ask-sdk has a dependency on boto3 which pulls in a whole lot of other packages. However the AWS Lambda runtime environment provides boto3 and there is no need to package that and its dependencies with our code. I do have boto3 installed in the system and import boto3 works, so I thought pip should be happy, but because of -t dist/ it always installs it.

Can I somehow install just ask-sdk and its dependencies that don't exist in the system, e.g. ask-sdk-core, but not those that are already installed?

One way is to list all the modules and use pip --no-deps but that means a constantly keeping track of the dependencies manually, we would like to avoid that.

Somehow I would like to tell pip: if the package is already installed, even if not in -t dist/ don't put a copy in dist/.

Is that possible?

Thanks!

Upvotes: 18

Views: 31513

Answers (3)

raychi
raychi

Reputation: 2663

Although you can't tell pip to "install all dependencies except those required by boto3", you can generate the needed requirements.txt by computing the difference between boto3 and ask-sdk from pip freeze output (tested with Python 3.6.6): 

# get boto3 requirements
pip install boto3 -t py_lib.boto3
PYTHONPATH=py_lib.boto3 pip freeze > requirements-boto3.txt

# get ask-sdk requirements
pip install ask-sdk -t py_lib.ask-sdk
PYTHONPATH=py_lib.ask-sdk pip freeze > requirements-ask-sdk.txt

# compute their difference
grep -v -x -f requirements-boto3.txt requirements-ask-sdk.txt > requirements-final.txt

# patch to add one missing dep from boto3
# aws don't have this for some reason
grep urllib3 requirements-boto3.txt >> requirements-final.txt

The requirements-final.txt contains the following:

ask-sdk==1.5.0
ask-sdk-core==1.5.0
ask-sdk-dynamodb-persistence-adapter==1.5.0
ask-sdk-model==1.6.2
ask-sdk-runtime==1.5.0
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1

To install the final set of dependencies to a folder:

pip install --no-deps -r requirements-final.txt -t py_lib

By skipping the boto3 dependencies, you can save about 45M of data from your python dependencies. The ask-sdk dependencies are only about 7.5M (2.1M compressed), allow you to use the build-in lambda code editor if you need to.

Upvotes: 13

Harshit Malhotra
Harshit Malhotra

Reputation: 74

This will work

$ pip install -t dist --no-deps ask-sdk

After the above command I checked out the dist directory content with tree and it had installed only ask-sdk without its dependencies

dist/
├── ask_sdk
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── __version__.py
│   ├── __version__.pyc
│   ├── standard.py
│   └── standard.pyc
└── ask_sdk-0.1.3.dist-info
    ├── INSTALLER
    ├── METADATA
    ├── RECORD
    ├── WHEEL
    └── top_level.txt

Upvotes: 2

Jim Todd
Jim Todd

Reputation: 1588

You can try the option

  --no-dependencies

To ignore all dependencies.

To exclude specific, you can put it in requirements file and pass it:

pip install --no-deps -r requirements.txt

Upvotes: 5

Related Questions