CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-storage
Aseem
Aseem

Reputation: 6787

Generate presigned url for uploading file to google storage using python

I want to upload a image from front end to google storage using javascript ajax functionality. I need a presigned url that the server would generate which would provide authentication to my frontend to upload a blob. How can I generate a presigned url when using my local machine.

Previously for aws s3 I would do :

pp = s3.generate_presigned_post(
            Bucket=settings.S3_BUCKET_NAME,
            Key='folder1/' + file_name,  
            ExpiresIn=20  # seconds
        )

When generating a signed url for a user to just view a file stored on google storage I do :

    bucket = settings.CLIENT.bucket(settings.BUCKET_NAME)
    blob_name = 'folder/img1.jpg'
    blob = bucket.blob(blob_name)
    url = blob.generate_signed_url(
        version='v4',
        expiration=datetime.timedelta(minutes=1),
        method='GET')

Upvotes: 6

Views: 5838

Answers (3)

Aseem
Aseem

Reputation: 6787

  • Create a service account private key and store it in SecretManager (SM).
  • In settings.py retrieve that key from SecretManager and store it in a constant - SV_ACCOUNT_KEY
  • Override Client() class func from_service_account_json() to take json key content instead of a path to json file. This way we dont have to have a json file in our file system (locally, cloudbuild or in GAE). we can just get private key contents from SM anytime anywhere.

settings.py

secret = SecretManager()
SV_ACCOUNT_KEY = secret.access_secret_data('SV_ACCOUNT_KEY')

signed_url_mixin.py

import datetime
import json

from django.conf import settings
from google.cloud.storage.client import Client
from google.oauth2 import service_account


class CustomClient(Client):
    @classmethod
    def from_service_account_json(cls, json_credentials_path, *args, **kwargs):
        """
        Copying everything from base func (from_service_account_json). 
        Instead of passing json file for private key, we pass the private key
        json contents directly (since we cannot save a file on GAE).
        Since its not properly written, we cannot just overwrite a class or a
        func, we have to rewrite this entire func.
        """
        if "credentials" in kwargs:
            raise TypeError("credentials must not be in keyword arguments")
        credentials_info = json.loads(json_credentials_path)
        credentials = service_account.Credentials.from_service_account_info(
            credentials_info
        )
        if cls._SET_PROJECT:
            if "project" not in kwargs:
                kwargs["project"] = credentials_info.get("project_id")

        kwargs["credentials"] = credentials
        return cls(*args, **kwargs)


class _SignedUrlMixin:
    bucket_name = settings.BUCKET_NAME
    CLIENT = CustomClient.from_service_account_json(settings.SV_ACCOUNT_KEY)
    exp_min = 4  # expire minutes

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.bucket = self.CLIENT.bucket(self.bucket_name)

    def _signed_url(self, file_name, method):
        blob = self.bucket.blob(file_name)
        signed_url = blob.generate_signed_url(
            version='v4',
            expiration=datetime.timedelta(minutes=self.exp_min),
            method=method
        )
        return signed_url


class GetSignedUrlMixin(_SignedUrlMixin):
    """
    A GET url to view file on CS
    """

    def get_signed_url(self, file_name):
        """
        :param file_name: name of file to be retrieved from CS.
            xyz/f1.pdf
        :return: GET signed url
        """
        method = 'GET'
        return self._signed_url(file_name, method)


class PutSignedUrlMixin(_SignedUrlMixin):
    """
    A PUT url to make a put req to upload a file to CS
    """

    def put_signed_url(self, file_name):
        """
        :file_name: xyz/f1.pdf
        """
        method = 'PUT'
        return self._signed_url(file_name, method)

Upvotes: 0

Aseem
Aseem

Reputation: 6787

Spent 100$ on google support and 2 weeks of my time to finally find a solution.

client = storage.Client() # works on app engine standard without any credentials requirements

But if you want to use generate_signed_url() function then you need service account Json key.

Every app engine standard has a default service account. ( You can find it in IAM/service account). Create a key for that default sv account and download the key ('sv_key.json') in json format. Store that key in your Django project right next to app.yaml file. Then do the following :

from google.cloud import storage
CLIENT = storage.Client.from_service_account_json('sv_key.json')
bucket = CLIENT.bucket('bucket_name_1')
blob = bucket.blob('img1.jpg') # name of file to be saved/uploaded to storage
pp = blob.generate_signed_url(
    version='v4',
    expiration=datetime.timedelta(minutes=1),
    method='POST')

This will work on your local machine and GAE standard. WHen you deploy your app to GAE, sv_key.json also gets deployed with Django project and hence it works.

Hope it helps you.

Upvotes: 13

bhito
bhito

Reputation: 2673

Editing my answer as I didn't understand the problem you were facing.

Taking a look at the comments thread in the question, as @Nick Shebanov stated, there's one possibility to accomplish what are you trying to when using GAE with flex environment.

I have been trying to do the same with GAE Standard environment with no luck so far. At this point, I would recommend opening a feature request at the public issue tracker so this gets somehow implemented.

Upvotes: 1

Related Questions