CODEWITHSUNDEEP
google-app-enginegoogle-cloud-storage
Evan Ruff
Evan Ruff

Reputation: 602

Location of GS File in Local/Dev AppEngine

I'm trying to trouble shoot some issues I'm having with an export task I have created. I'm attempting to export CSV data using Google Cloud Storage and I seem to be unable to export all my data. I'm assuming it has something to do with the (FAR TOO LOW) 30 second file limit when I attempt to restart the task.

I need to trouble shoot, but I can't seem to find where my local/development server writing the files out. I see numerous entries in the GsFileInfo table so I assume something is going on, but I can't seem to find the actual output file.

Can someone point me to the location of the Google Cloud Storage files in the local AppEngine development environment?

Thanks!

Upvotes: 4

Views: 2113

Answers (2)

Evan Ruff
Evan Ruff

Reputation: 602

As tkaitchuck mentions above, you can use the included LocalRawGcsService to pull the data out of the local.db. This is the only way to get the file, as they are stored in the Local DB using the blobstore. Here's the original answer:

which are the files uri on GAE java emulating cloud storage with GCS client library?

Upvotes: 1

Yey
Yey

Reputation: 554

Looking at dev_appserver code, looks like you can specify a path or it will calculate a default based on the OS you are using.

 blobstore_path = options.blobstore_path or os.path.join(storage_path,
                                                        'blobs')

Then it passed this path to blobstore_stub (GCS storage is backed by blobstore stub), which seems to shard files by their blobstore key.

  def _FileForBlob(self, blob_key):
"""Calculate full filename to store blob contents in.

This method does not check to see if the file actually exists.

Args:
  blob_key: Blob key of blob to calculate file for.

Returns:
  Complete path for file used for storing blob.
"""
    blob_key = self._BlobKey(blob_key)
    return os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:])

For example, i'm using ubuntu and started with dev_appserver.py --storage_path=~/tmp, then i was able to find files under ~/tmp/blobs and datastore under ~/tmp/datastore.db. Alternatively, you can go to local admin_console, the blobstore viewer link will also display gcs files.

Upvotes: 1

Related Questions