CODEWITHSUNDEEP
scrapy
user6712306
user6712306

Reputation: 57

Crawl website every x minutes/seconds using scrapy

I'm crawling a blog using Scrapy using the command "scrapy crawl posts". I need to check for new posts every 5 minutes. How can I do it?

Thanks in advance.

Upvotes: 0

Views: 1200

Answers (1)

user378704
user378704

Reputation:

Edit your /etc/crontab file to include this line. It will setup your script to run every 5 minutes.

*/5 *   * * *   unix-username   cd /your/path/to/scrapy-project/;scrapy crawl posts

For windows you can take a look at the answer mentioned here

Upvotes: 1

Related Questions