GAE TaskQueue: Sample code for accessing pull queue from outside App Engine?

Question

I'm attempting to use GAE TaskQueue's REST API to pull tasks from a queue to an external server (a server not on GAE).

• Is there a library that does this for me?

• The API is simple enough, so I just need to figure out authentication. I examined the request sent by gtaskqueue_sample from google-api-python-client using --dump_request and found the authorization: OAuth XXX header. Adding that token to my own requested worked, but the token seems to expire periodically (possibly daily), and I can't figure out how to re-generate it. For that matter, gtaskqueue_sample itself no longer works (the call to https://accounts.google.com/o/oauth2/token fails with No JSON object could be decoded).

How does one take care of authentication? This is a server app so ideally I could generate a token that I could use from then on.

Answer 1

This question is old, but it still applies, so I am going to attempt a better answer based on my recent experience.

It is possible to access pull task queues outside of appengine but as the asker stated, there are no good examples, so here is a more in depth guide. In my case I had a custom python script that needed to poll the queue for new jobs to run.

Before taking this route, you also have the option of rolling your own security and making a simple web wrapper to the appengine taskqueue calls. I was tempted to go that route after dealing with this, but since this is working I'm using it for now.

Setup Your Machine

• pip install --upgrade google-api-python-client
• easy_install python-gflags

Setup Your Account

• Using Google Cloud Console, create a Registered App (if you don't already have one. Click on your AppEngine project -> API's and auth -> Registered Apps. You can enter a name and application type and then accept the defaults. Once it is created, note the Client Id and Client Secret for later.

• Also Update your Consent Screen (API's and auth -> Consent Screen). Note that you will only need this consent screen to setup your oauth credentials for the first time. You will need to enter Email Address and a Product Name (I entered a HomePage Url also).

Generate OAuth Credentials

• You only need to generate a credentials file once, then it will be used for future calls in your python script. Run this python code which opens a browser and generates the credential file. A reference for this code is here.

  from oauth2client.tools import run
  from oauth2client.client import OAuth2WebServerFlow
  from oauth2client.file import Storage
  import gflags
  FLAGS = gflags.FLAGS
  
  storage = Storage('credentials.json')
  
  flow = OAuth2WebServerFlow(client_id='<your_client_id>',
                         client_secret='<your_client_secret>',
                         scope='https://www.googleapis.com/auth/taskqueue',
                         redirect_uri='urn:ietf:wg:oauth:2.0:oob')
  
  credentials = run(flow, storage )
  

Make Your Taskqueue Calls

• Make sure you have added a pull queue in your AppEngine queue.yaml, with the email address you used in the oauth step above.

  from oauth2client.tools import run
  from oauth2client.client import OAuth2WebServerFlow
  from oauth2client.file import Storage
  from apiclient.discovery import build
  import httplib2
  
  storage = Storage('credentials.json')
  credentials = storage.get()
  http = httplib2.Http()
  http = credentials.authorize(http)
  task_api = build('taskqueue', 'v1beta2')
  tasks = task_api.tasks().lease(project='<your appengine project>',taskqueue='<pull queue name>', numTasks=1, leaseSecs=600).execute(http=http)
  task = tasks['items'][0]
  payload = task['payloadBase64']
  payload = base64.b64decode(payload)
  
  #then do your work and delete the task when done
  
  task_api.tasks().delete(project='s~<your appengine project>',taskqueue='<pull queue name>', task=task['id']).execute(http=http)
  

• Task Queue API Reference

• Note the prefix 's~' in front of the project name in the delete call. It would only work if I added this and I believe it is a bug.

---

Update 7/1/2014

So there is actually an easier way to make server to server calls. This way doesn't require you using the "flow" (logging on to google) to get an access key.

Setup Your Machine

• pip install --upgrade google-api-python-client
• pip install pyOpenSSL

Setup Your Account

• Using Google Cloud Console, create a Registered App (if you don't already have one. Click on your AppEngine project -> API's & Auth -> Credentials. Click Create New Client ID, specify Service Account, then click Create Client ID. A download box will pop up to download your private key, save this to your code directory (or wherever, I saved as client_key.p12). On the web interface, note the Client ID and Email.

Replace the Credential Code from Above

from oauth2client.client import SignedJwtAssertionCredentials

email = '<***>.gserviceaccount.com'
f = file('client_key.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(email,
                                            key,
                                            scope='https://www.googleapis.com/auth/taskqueue')

Answer 2

These APIS work only for GAE server since the queues can be created only via queue.yaml and infact API does not expose any API for inserting queue and tasks or project.

Answer 3

The pull queues page has a whole section about client libraries and sample code.