CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-storage
Ricardo Pereira
Ricardo Pereira

Reputation: 1

GAE Can not use cloudstorage on dev environment

I'm using the dev build of GAE to upload some zip files, unzip and the edit them on the screen, as of now i'm just trying to see if the files can be opened after the upload and unzip.

If I try to open it any other place them on a background tasks i get the following error:

Traceback (most recent call last):
  File "/home/ricardo/workspace/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 1302, in communicate
    req.respond()
  File "/home/ricardo/workspace/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 831, in respond
    self.server.gateway(self).respond()
  File "/home/ricardo/workspace/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 2115, in respond
    response = self.req.server.wsgi_app(self.env, self.start_response)
  File "/home/ricardo/workspace/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 292, in __call__
    return app(environ, start_response)
  File "/home/ricardo/workspace/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/request_rewriter.py", line 314, in _rewriter_middleware
    response_body = iter(application(environ, wrapped_start_response))
  File "/home/ricardo/workspace/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 148, in __call__
    environ[http_runtime_constants.REQUEST_ID_ENVIRON])
KeyError: 'HTTP_X_APPENGINE_DEV_REQUEST_ID'

Here's the code:

class ExportConfiguration(blobstore_handlers.BlobstoreUploadHandler):
def post(self, store_id):
    # upload_files = self.get_uploads('file')
    bucket = '/' + BUCKET
    upload = self.get_uploads()[0]
    # write_retry_params = cloudstorage.RetryParams(backoff_factor=1.1)
    file = zipfile.ZipFile(upload.open())
    # logging.info(file)
    for name in file.namelist():
        outfile = file.read(name)
        outfile_list = name.split('/')
        for idx, path in enumerate(outfile_list):
            if path == 'bundles':
                node = outfile_list[idx - 1]
                outfile_path = 'data/{}/{}'.format(node, '/'.join(outfile_list[idx:]))
        params = {'path': '{}/{}'.format(bucket, outfile_path), 'file': outfile}

        task = Task(url='/tasks/write_loaders/',
                    params=params,
                    method='POST',
                    target='background-tasks')
        task.add()
        # outfile_full = '{}/{}'.format(bucket, outfile_path)
        # logging.info(outfile_full)
        # gcs_file = cloudstorage.open(outfile_full, 'w', options={'x-goog-meta-foo': 'foo', 'x-goog-meta-bar': 'bar'}, retry_params=write_retry_params)
        # gcs_file.write('outfile')
        # gcs_file.close()
    # file.extract()
    # file = str(self.request.POST.get(' filename'))
    # open(str(self.request.POST.get(' filename')).split('"')[1])
    # blob_info = upload_files[0]
    stats = cloudstorage.listbucket(bucket)
    for s in stats:
        logging.info(s.filename)
    logging.info(dir(stats))
    resp = {'code': 0, 'result': 'Success'}
    self.response.headers['Content-Type'] = 'application/json'
    self.response.out.write(json.dumps(resp, separators=(',', ':')))


class LoaderWriter(webapp2.RequestHandler):
    def post(self):
        path = self.request.POST.get('path')
        file = self.request.POST.get('file')
        file = file.encode('utf-8')
        file = etree.XML(file)
        gcs_file = cloudstorage.open(path, 'w')
        gcs_file.write(etree.tostring(file))
        gcs_file.close()

I tried to see if this was a gcs configuration problem, to no avail, has anyone else run into the same problem ?

Upvotes: 0

Views: 93

Answers (2)

Dan Cornilescu
Dan Cornilescu

Reputation: 39814

Potential approach based on @Peuchele's answer.

In my case I was able to work around a similar problem caused by the differences between blobstore_handlers.BlobstoreUploadHandler and webapp2.RequestHandler by using double inheritance. Something like:

class ExportConfiguration(webapp2.RequestHandler,
                          blobstore_handlers.BlobstoreUploadHandler):

or (the inheritance order may matter):

class ExportConfiguration(blobstore_handlers.BlobstoreUploadHandler,
                          webapp2.RequestHandler):

Note that this approach might not work in all cases, depending on the handler's implementation.

Upvotes: 0

Peuchele
Peuchele

Reputation: 346

The problem seems to be related to the fact that you are extending BlobstoreUploadHandler. Apparently, this handler strips http headers like HTTP_X_APPENGINE_DEV_REQUEST_ID which are required for further API calls performed by gcs library.

If you try to open the file from a regular handler webapp2.RequestHandler it should work.

Upvotes: 1

Related Questions