CODEWITHSUNDEEP
google-colaboratorykaggle
Bohrium272
Bohrium272

Reputation: 726

Setting environment variables in Google Colab

I'm trying to use the Kaggle CLI API, and in order to do that, instead of using kaggle.json for authentication, I'm using environment variables to set the credentials.

!pip install --upgrade kaggle

!export KAGGLE_USERNAME=abcdefgh
!export KAGGLE_KEY=abcdefgh

!export -p

However, the printed list of env. variables doesn't contain the ones I set above.

declare -x CLICOLOR="1"
declare -x CLOUDSDK_CONFIG="/content/.config"
declare -x COLAB_GPU="1"
declare -x CUDA_PKG_VERSION="9-2=9.2.148-1"
declare -x CUDA_VERSION="9.2.148"
declare -x CUDNN_VERSION="7.4.1.5"
declare -x DATALAB_SETTINGS_OVERRIDES="{\"kernelManagerProxyPort\":6000,\"kernelManagerProxyHost\":\"172.28.0.3\",\"jupyterArgs\":[\"notebook\",\"-y\",\"--no-browser\",\"--log-level=DEBUG\",\"--debug\",\"--NotebookApp.allow_origin=\\\"*\\\"\",\"--NotebookApp.log_format=\\\"%(message)s\\\"\",\"--NotebookApp.disable_check_xsrf=True\",\"--NotebookApp.token=\",\"--Session.key=\\\"\\\"\",\"--Session.keyfile=\\\"\\\"\",\"--ContentsManager.untitled_directory=\\\"Untitled Folder\\\"\",\"--ContentsManager.untitled_file=\\\"Untitled File\\\"\",\"--ContentsManager.untitled_notebook=\\\"Untitled Notebook\\\"\",\"--KernelManager.autorestart=True\",\"--ip=\\\"172.28.0.2\\\"\"]}"
declare -x DEBIAN_FRONTEND="noninteractive"
declare -x ENV="/root/.bashrc"
declare -x GIT_PAGER="cat"
declare -x GLIBCPP_FORCE_NEW="1"
declare -x GLIBCXX_FORCE_NEW="1"
declare -x HOME="/root"
declare -x HOSTNAME="2ced809e9844"
declare -x JPY_PARENT_PID="57"
declare -x LANG="en_US.UTF-8"
declare -x LD_LIBRARY_PATH="/usr/lib64-nvidia"
declare -x LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4"
declare -x MPLBACKEND="module://ipykernel.pylab.backend_inline"
declare -x NCCL_VERSION="2.3.7"
declare -x NVIDIA_DRIVER_CAPABILITIES="compute,utility"
declare -x NVIDIA_REQUIRE_CUDA="cuda>=9.2"
declare -x NVIDIA_VISIBLE_DEVICES="all"
declare -x OLDPWD="/"
declare -x PAGER="cat"
declare -x PATH="/usr/local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin"
declare -x PWD="/content"
declare -x PYTHONPATH="/env/python"
declare -x SHELL="/bin/bash"
declare -x SHLVL="2"
declare -x TERM="xterm-color"
declare -x TF_FORCE_GPU_ALLOW_GROWTH="true"
declare -x _="/tools/node/bin/forever"
declare -x __EGL_VENDOR_LIBRARY_DIRS="/usr/lib64-nvidia:/usr/share/glvnd/egl_vendor.d/"

Upvotes: 57

Views: 85299

Answers (3)

korakot
korakot

Reputation: 40888

If you like %magic, you can also use %env to make it a bit shorter.

%env KAGGLE_USERNAME=abcdefgh

If the value is in a variable you can also use

%env KAGGLE_USERNAME=$username

For automatic setting env from Colab Secret, you can use this library.

!pip install set-env-colab-kaggle-dotenv
from set_env import set_env
set_env("GOOGLE_API_KEY")
set_env("WANDB_API_KEY")

Upvotes: 95

Justahelper
Justahelper

Reputation: 19

You could either use intput or simply getpass to set env variables.(Assuming yr using JP noteb)

!pip install getpass

import getpass
import os


def SetEnv(name):
    secret = getpass.getpass(f"Enter value of {name}: ")
    os.environ[name] = secret

# to get var
mys = os.environ['secret']
print(mys)

# Usage to set var
SetEnv("api_id")
SetEnv("api_hash")
SetEnv("bot_token")

Upvotes: 1

Bob Smith
Bob Smith

Reputation: 38639

I think you want to do something like:

import os
os.environ['KAGGLE_USERNAME'] = ...
os.environ['KAGGLE_KEY'] = ...

The reason is that !export will assign the environment variable in an ephemeral sub-shell. But, you want to update the environment for the Python subprocess that spawns those sub-shells.

enter image description here

Upvotes: 67

Related Questions