CODEWITHSUNDEEP
pythondjangoweb-scrapingscrapy
PXY
PXY

Reputation: 61

How to run scrapy spider with arguments inside django view

The user could input keyword in a form and submit it, so I can get the keyword in the views. Then I can make the start_url with the keyword. How can I pass the start_url to a scrapy spider and start it?

This is my view method.

def results(request):
    """Return the search results"""
    key= request.GET['keyword'].strip()
    books = Book.objects.filter(title__contains=key)
    if books is None:
        # I want to call the scrapy spider here.
        pass
        books = Book.objects.filter(title__contains=key)
    context = {'books': books, 'key': title}
    return render(request, 'search/results.html', context)

This is the init() method of my spider class. 

def __init__(self, key):
    self.key = key
    url = "http://search.example.com/?key=" + key
    self.start_urls = [url]

Upvotes: 6

Views: 3122

Answers (1)

Tigrou
Tigrou

Reputation: 201

this worked for me :

from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
if books is None:
    # I want to call the scrapy spider here.
    os.environ.setdefault("SCRAPY_SETTINGS_MODULE","whereyourscrapysettingsare")
crawler_settings = get_project_settings()
crawler = CrawlerRunner(crawler_settings)
crawler.crawl(yourspider, key=key)

from http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

Upvotes: 6

Related Questions