CODEWITHSUNDEEP
cmdscheduled-tasksscrapy
sulav_lfc
sulav_lfc

Reputation: 782

Schedule scrapy command scrapy crawl

I've built a crawler to crawl and extract links matching a fixed regex. Now i want to schedule the task in windows task scheduler, such that it executes the crawler using command line prompt scrapy crawl crawlername. Any help would be useful?

Upvotes: 3

Views: 1219

Answers (2)

Ibrahim Mlata
Ibrahim Mlata

Reputation: 29

You can create a fil like run.py

and put your script like that example with multiple spiders :

import os
import logging

def crawl():
    os.system('scrapy crawl spider1')
    os.system('scrapy crawl spider2')
    os.system('scrapy crawl spider3')

schedule.every(5).minutes.do(crawl)
while True:
    schedule.run_pending()```

Upvotes: 0

bryant1410
bryant1410

Reputation: 6340

You can create a bat file with the content:

cd path_to_scrapy_project
scrapy crawl crawlername

Then schedule this script with Windows Task Scheduler.

Upvotes: 2

Related Questions