CODEWITHSUNDEEP
amazon-web-servicescondaamazon-sagemaker
salhin
salhin

Reputation: 2652

Make custom kernels available to SageMaker Studio notebooks

On starting the SageMaker Studio server, I can only see a set of predefined kernels when I select kernel for any notebook.

Predefined Kernels

I create conda environments and persist them between sessions by pointing .condarc to a custom miniconda directory stored on EFS.

I want all notebooks to have access to environments stored in the custom miniconda directory. I can do that on the system terminal but can't seem to find a way to make the kernels available to notebooks.

I am aware of Life Cycle Configuration but that seems to be working only with notebooks instances rather than SageMaker Studio.

Desired outcomes

Ideally making custom kernels persistently available to notebooks but if that isn't feasible or requires custom docker image, I am happy with running a script manually every time I run the server.

What I have tried so far:

I ran the following which is a tweaked version of start.sh meant to be for Life Cycle Configuration.

#!/bin/bash

set -e

sudo -u sagemaker-user -i <<'EOF'
unset SUDO_UID
WORKING_DIR=/home/sagemaker-user/.SageMaker/custom-miniconda/
source "$WORKING_DIR/miniconda/bin/activate"
for env in $WORKING_DIR/miniconda/envs/*; do
    BASENAME=$(basename "$env")
    source activate "$BASENAME"
    python -m ipykernel install --user --name "$BASENAME" --display-name "$BASENAME"
done

EOF

That didn't work and I couldn't access the kernels from the notebooks.

Upvotes: 2

Views: 5006

Answers (1)

Kraken10
Kraken10

Reputation: 33

If you need a persistent custom kernel in SageMaker studio, you can create an ECR repository and build a docker image with custom environment configurations. This image can then be attached to the SageMaker studio notebooks. Reference link!

SageMaker studio now also supports the use of lifecycle configurations. Reference link!

Upvotes: 1

Related Questions