CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Akila
Akila

Reputation: 97

Run Python Scrapy script via HTTP request

I'm looking for an example to run scrapy script via HTTP request. I'm planing to send url as a parameter that i need to crawl, via GET or POST method. How can i do that.

Upvotes: 0

Views: 367

Answers (2)

user10600066
user10600066

Reputation: 53

Try something like that.

from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from testspiders.spiders.followall import FollowAllSpider
from scrapy.utils.project import get_project_settings

spider = FollowAllSpider(domain='url.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()

Upvotes: 0

Biswanath
Biswanath

Reputation: 9185

You should use scrapyd.

Link to the GitHub project page.

Once you are using scrapyd you can use this api to scedule a crawl.

Upvotes: 3

Related Questions