CODEWITHSUNDEEP
pythongoogle-app-enginecronarguments
user708517
user708517

Reputation:

How to pass arguments to a python cron task on Google App Engine?

I have an application written in Python. It collects and parses data from certain sources (RSS, Atom, Twitter, etc.) and if new data is found, it's saved to a MySQL database.

I have many sources defined on the application itself, and not all can be checked on the same time basis because of restrictions (Twitter and its 350 queries per hour, for example), so the idea is to cron them separately by passing sources as arguments.

myscript.py --update --source ABC (Fetch, parse, check and save data from ABC source) myscript.py --update --source XYZ (Fetch, parse, check and save data from XYZ source)

So far this point, I got that working fine on my local cron, but I'd like to get this running on Google App Engine, with something like this:

cron.yaml:

cron:
- description: update source ABC
  url: /myscript.py --source ABC
  schedule: every 5 minutes
  login: admin
- description: update source XYZ
  url: /myscript.py --source XYZ
  schedule: every 12 minutes
  login: admin

I haven't found any way to get this done on App Engine Cron documentation. Any workarounds?

Upvotes: 4

Views: 1386

Answers (2)

Nick Johnson
Nick Johnson

Reputation: 101149

Since cron.yaml can only be updated by uploading a new version of your app, and has an upper limit on the number of entities, you'd be better off solving this by writing your app to use the task queue to schedule operations.

Upvotes: 0

Kevin P
Kevin P

Reputation: 1655

Put the source variable in your url path:

cron:
- description: update source ABC
  url: /cron/ABC/
  schedule: every 5 minutes
  login: admin
- description: update source XYZ
  url: /cron/XYZ/
  schedule: every 12 minutes
  login: admin

webapp.WSGIApplication([('/cron/([^/]+)/', CronHandler)])    

class CronHandler(webapp.RequestHandler):
    def post(self, source): 
            #do something with source variable...

Upvotes: 3

Related Questions