CODEWITHSUNDEEP
pythoncloud
Rajesh
Rajesh

Reputation: 53

Python, Cloud - Online Web Scraping Tool with scheduling capabilities

I'm planning to develop a Web/cloud application using python which does the following,
1. Upload Perl/Python scraping scripts and execute.
2. Uploaded scripts to run on a Schedule.
3. Run multiple instance of same script with different input parameters.
4. Measure scripts performance.
5. View Scripts standard output/log.

I got limited/no knowledge about technologies which can satisfy my requirement,
Tips, Pointers, Ideas to existing work, libraries, open source implementation etc are appreciated.

Thanks,
Rajesh.

Upvotes: 1

Views: 1496

Answers (1)

Mikko Ohtamaa
Mikko Ohtamaa

Reputation: 83418

Use Scrapy as the base for your work:

http://scrapy.org/

For uploading, script performance and output (through web interface, I assume) you need to write your custom web front end which stores this information in a database where you can then explore it. One option for this is Pyramid:

http://pypi.python.org/pypi/pyramid/

For Python cloud deployments see Heroku:

http://www.heroku.com/

Upvotes: 1

Related Questions