CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-functionskagglegoogle-secret-manager
Sean MacInnis
Sean MacInnis

Reputation: 1

How do I include the Kaggle api key into a google cloud function so that the Kaggle api can find it?

I'm trying to write a google cloud function that automatically downloads data from a certain kaggle data set and upload it to a google cloud bucket. No matter how I try to include my kaggle api key in my google cloud function, the kaggle api can't find it!

I uploaded my kaggle api key to the secret manager and I wrote some code to write the key value into a file stored in /tmp, and set KAGGLE_CONFIG_DIR to /tmp. I granted the cloud function secret manager admin permissions and created added a runtime service account with permission to access the kaggle key as well just to make sure

I reference the secret via a mount path that I set here

enter image description here

First I tried just setting KAGGlE_CONFIG_DIR to the mount path, but that didn't work:

os.environ["KAGGLE_CONFIG_DIR"] = "/.kaggle"

Then I tried to write the key to /tmp, and set KAGGlE_CONFIG_DIR to /tmp and that didn't work either.

def get_kaggle_api_key():
    kaggle_key_path = "/.kaggle/kaggle.json"  

    #read the kaggle key
    with open(kaggle_key_path, 'r') as f:
        kaggle_api_key = f.read().strip()

    #save the key to /tmp/
    kaggle_config_path = "/tmp/kaggle.json" 
    with open(kaggle_config_path, 'w') as f:
        json.dump(json.loads(kaggle_api_key), f)

    os.environ["KAGGLE_CONFIG_DIR"] = "/tmp"

I always get this error:

OSError: Could not find kaggle.json. Make sure it's located in /www-data-home/.config/kaggle. Or use the environment method. See setup instructions at https://github.com/Kaggle/kaggle-api/

The setup instructions don't help, they just say to set KAGGLE_CONFIG_DIR to where the key is stored, which I did.

This is my first time trying to write a cloud function to access an api, and I'm not really sure what I'm doing. Please help!

Upvotes: 0

Views: 72

Answers (1)

DazWilkin
DazWilkin

Reputation: 40326

I'm not a Kaggle user but hopefully this example will help.

There are 3 (possibly 2) steps but importantly you must create a volume and then create a volume mount:

  1. Create an environment variable for KAGGLE_CONFIG_DIR
  2. Create a Volume (e.g. KAGGLE-CONFIG-DIR) using the Secret
  3. Create a Volume Mount (e.g. /.kaggle)

Set an environment variable (KAGGLE_CONFIG_DIR) to /.kaggle:

Environment Variable

Create a Volume:

Volume

Create a Container Volume Mount:

Volume Mount

And some Python:

@functions_framework.http
def root(request):
    # Get the value from the environment
    kaggle_config_dir = os.getenv("KAGGLE_CONFIG_DIR")

    # Open the file
    with open(f"{kaggle_config_dir}/kaggle.json", 'r') as file:
        content = file.read()
        # Return the content
        return f"{kaggle_config_dir}/kaggle.json: {content}:

Yields:

.kaggle/kaggle.json: {"your":"JSON"}

Upvotes: 1

Related Questions