CODEWITHSUNDEEP
amazon-sagemaker
Thanakrit Boonquarmdee
Thanakrit Boonquarmdee

Reputation: 1

List of Python library in Sagemaker SKlearnProcessor

Is there a way to list the Python library in Sagemaker SKlearnProcessor? Could not find the list from official doc. , https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html#data-pre-processing-and-model-evaluation-with-scikit-learn

Upvotes: 0

Views: 1217

Answers (1)

Abdelrahman Maharek
Abdelrahman Maharek

Reputation: 872

I couldn't find the list of installed packages.

To work around this issue, you can refer to the GitHub example and make modifications to the preprocessing.py file to print the list of installed libraries. One approach is to utilize the command os.system('pip freeze') and then inspect the CloudWatch logs for the respective job.

I've tested this with the sagemaker-scikit-learn:0.20.0-cpu-py3 environment, and below is the complete list of installed packages:

bcrypt==3.2.0
boto3==1.17.36
botocore==1.20.36
brotlipy==0.7.0
certifi==2020.6.20
cffi @ file:///tmp/build/80754af9/cffi_1613246939562/work
chardet @ file:///tmp/build/80754af9/chardet_1607706768982/work
click==7.1.2
conda==4.9.2
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1603018138503/work
cryptography @ file:///tmp/build/80754af9/cryptography_1615530759576/work
Flask==1.1.1
gevent==21.1.2
greenlet==1.0.0
gunicorn==20.0.4
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
inotify-simple==1.2.1
itsdangerous==1.1.0
Jinja2==2.11.3
jmespath==0.10.0
MarkupSafe==1.1.1
mlio==0.1.3
numpy==1.20.1
pandas==0.25.3
paramiko==2.7.2
protobuf==3.15.6
psutil==5.8.0
pyarrow==0.14.1
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyNaCl==1.4.0
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
python-dateutil==2.8.1
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
retrying==1.3.3
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016701961/work
s3transfer==0.3.6
sagemaker-containers==2.8.6.post2
sagemaker-sklearn-container @ file:///sagemaker_sklearn_container-1.0-py2.py3-none-any.whl
sagemaker-training==3.7.3
scikit-learn==0.20.0
scipy==1.6.2
six @ file:///tmp/build/80754af9/six_1605205313296/work
tqdm @ file:///tmp/build/80754af9/tqdm_1615925068909/work
typing==3.7.4.3
urllib3 @ file:///tmp/build/80754af9/urllib3_1615837158687/work
Werkzeug==1.0.1
zope.event==4.5.0
zope.interface==5.3.0

Additionally, you can install extra libraries using the command os.system('pip install ...').

Update 1/11/2023: The list of installed packages during container build can be found in the requirements.txt file, but it may not include the entire list.

Upvotes: 3

Related Questions