How do I get a valid blobstore key for a google cloudstorage object that is generated dynamically?

Question

I have a use case where I need to dynamically generate a csv/txt file and then save the file's Key to the datastore so the file can be downloaded at a later time. I seem to be having a problem generating a valid key to use in conjuction with Ferris' download uri. So for example:

import cloudstorage
from google.appengine.ext import blobstore
@route
def make_file(self):
    # Hardcoded filename, this will overwrite prior file if it exists
    filename = '/mydomain.appspot.com/some_folder/myawesometextfile2.txt'

    # Create file 
    gcs_file = cloudstorage.open(filename,'w',content_type='text/plain')

    # Generate the file's contents (pretend this is being done dynamically)
    gcs_file.write('Doe,John\n')
    gcs_file.write('Smith,Jane\n')

    # Close the file
    gcs_file.close() 

    # This is supposed to create a blobkey that represents the cloud object
    psuedo_blobkey = blobstore.create_gs_key('/gs'+filename)

    # This is supposed to also create a blobkey...I think?
    another_key = blobstore.BlobKey(psuedo_blobkey)

    # Here I attempt to store this in the datastore.
    new_file = self.meta.Model(
                        file_key = another_key,
                        file_name_actual = filename,
                        )
    new_file.put() 

If I try to save "psuedo_blobkey" into my NDB model I get an error something like "Expected BlobKey, got AMIfv-yadda-yadda-yadda".

If I try to save "another_key" into my model it stores the key with no problem, but when I try to access the entity via the datastore viewer in the appengine dashboard it tells me that it's not a valid key. As such when I try to use the key in my jinja template like so:

<a href="{{uri("download", blob=file_key)}}" target="_blank">Export</a>

Ferris renders the error "The resource could not be found." which makes sense because apparently its not a valid key.

So I guess my question is, how in the world do I get a valid key for a file I've generated dynamically in google cloud storage?

BTW: getting the key via an upload operation is easy but for some reason a GCS object that is generated dynamically does not produce the same result.

Thanks in advance

Answer 1

OK finally figured it out after reading this post and modifying Kekito's example.

Ultimately what I was trying to do was allow the end-user to export data from the NDB datastore. I originally thought that I had to generate a file, put it in google cloud storage (GCS) or the blobstore, then provide a download URL. I turns out it is much simpler than that. I don't even think you need to create a file in GCS to do this (even though my example does add a file to GCS).

All I had to do was add the Content-Disposition as an attachment before returning the file. This generated the Save-As dialog box allowing the user to determine the download location. Without the content-disposition the CSV/TXT content was rendered on the browser and if the user tried to do a save-as in their browser the only option was HTML and it could not be changed (in Chrome anyway).

import csv
import cloudstorage
@route
def export_users(self):
    # This can be any NDB query
    user_data = self.Meta.Model.get_users(is_active=True)

    # Set the file name 
    filename = '/mydomain.appspot.com/user_list.csv'

    # Create the file
    gcs_file = cloudstorage.open(filename,'w',content_type='text/csv')

    # Setup the CSV file for writing
    writer = csv.writer(gcs_file, delimiter=',')

    # Loop thru the query and write each row to the CSV file
    for user_info in user_data:
        writer.writerow( (str(user_info.user_lname),str(user_info.user_fname)) )

    # Close the file
    gcs_file.close()

    try:
        # Open the file for reading
        with cloudstorage.open(filename, "r") as the_file:
            # This will open the Save-As dialog box so the end-user can choose where to save the file
            self.response.headers["Content-Disposition"] = "'attachment'; filename=user_list.csv"

            # Push the file to the browser
            return the_file.read(32*1024*1024).decode("utf-8")
    except cloudstorage.NotFoundError:
        return "it failed"  

Answer 2

I store filenames instead of keys. The files I store are not very big, so in my app engine code I read them like this:

def get_gcs(fn):
    "Slurps a GCS file and returns it as unicode."
    try:
        with gcs.open(fn, "r") as f:
            return f.read(32*1024*1024).decode("utf-8")
    except gcs.NotFoundError:
        return ""

And I then serve them to the user using standard app engine techniques. If your files are small, this works fine, but if your files are big there are probably better ways of doing this.

Since you are getting the files via app engine, this works even if the ACLs are set to private.