CODEWITHSUNDEEP
pythonflaskscrapyweb-crawler
Coding_Rabbit
Coding_Rabbit

Reputation: 1337

How to pass arguments when using Crawler Runner in Flask?

I've read the Official Document of scrapy -1.0.4 about how to run multiple spiders programmatically.It supplies a way to do this by Crawler Runner, so I use this in my Flask App.But there is a problem that I want to pass an argument to the Crawler to be part of the Start Urls. I don't know how to do this. Here is my Flask App code:

app.route('/search_process', methods=['GET'])
def search():
    configure_logging()
    runner = CrawlerRunner()
    runner.crawl(EPGDspider)
    # runner.crawl(GDSpider)
    d = runner.join()
    d.addBoth(lambda _: reactor.stop())

    reactor.run()
    return redirect(url_for('details'))

Here is my spiders code:

__author__ = 'Rabbit'
import scrapy
from scrapy.selector import Selector
from scrapy import Request
from scrapy import Item, Field

class EPGD(Item):

    genID = Field()
    genID_url = Field()
    taxID = Field()
    taxID_url = Field()
    familyID = Field()
    familyID_url = Field()
    chromosome = Field()
    symbol = Field()
    description = Field()

class EPGDspider(scrapy.Spider):
    name = "EPGD"
    allowed_domains = ["epgd.biosino.org"]
    term = "man"
    start_urls = ["http://epgd.biosino.org/EPGD/search/textsearch.jsp?textquery="+term+"&submit=Feeling+Lucky"]
    MONGODB_DB = name + "_" + term
    MONGODB_COLLECTION = name + "_" + term

    def parse(self, response):
        sel = Selector(response)
        sites = sel.xpath('//tr[@class="odd"]|//tr[@class="even"]')
        url_list = []
        base_url = "http://epgd.biosino.org/EPGD"

        for site in sites:
            item = EPGD()
            item['genID'] = map(unicode.strip, site.xpath('td[1]/a/text()').extract())
            item['genID_url'] = base_url+map(unicode.strip, site.xpath('td[1]/a/@href').extract())[0][2:]
            item['taxID'] = map(unicode.strip, site.xpath('td[2]/a/text()').extract())
            item['taxID_url'] = map(unicode.strip, site.xpath('td[2]/a/@href').extract())
            item['familyID'] = map(unicode.strip, site.xpath('td[3]/a/text()').extract())
            item['familyID_url'] = base_url+map(unicode.strip, site.xpath('td[3]/a/@href').extract())[0][2:]
            item['chromosome'] = map(unicode.strip, site.xpath('td[4]/text()').extract())
            item['symbol'] = map(unicode.strip, site.xpath('td[5]/text()').extract())
            item['description'] = map(unicode.strip, site.xpath('td[6]/text()').extract())
            yield item

        sel_tmp = Selector(response)
        link = sel_tmp.xpath('//span[@id="quickPage"]')

        for site in link:
            url_list.append(site.xpath('a/@href').extract())

        for i in range(len(url_list[0])):
            if cmp(url_list[0][i], "#") == 0:
                if i+1 < len(url_list[0]):
                    print url_list[0][i+1]
                    actual_url = "http://epgd.biosino.org/EPGD/search/"+ url_list[0][i+1]
                    yield Request(actual_url, callback=self.parse)
                    break
                else:
                    print "The index is out of range!"

As you can see,the term has been set in the code already now. I just want to pass the argument term from Flask App to my spiders and compose the start urls dynamically. Its effect kind of like the situation in this question:How to pass a user defined argument in scrapy spider.But all the things are done in Flask App programmatically, not by command line. But I don't know how to do this, can someone tell me how to deal with this?

Upvotes: 3

Views: 2084

Answers (1)

Coding_Rabbit
Coding_Rabbit

Reputation: 1337

I've solve this problem by crawl(crawler_or_spidercls, *args, **kwargs), you can pass arguments by this method. Here is my Flask App code:

def search():
    configure_logging()
    runner = CrawlerRunner()
    runner.crawl(EPGDspider, term="man")
    d = runner.join()
    d.addBoth(lambda _: reactor.stop())

    reactor.run()

And my spiders code(you can override the _init_ method and construct your dynamic start urls):

def __init__(self, term=None, *args, **kwargs):
        super(EPGDspider, self).__init__(*args, **kwargs)
        self.start_urls = ['http://epgd.biosino.org/EPGD/search/textsearch.jsp?textquery=%s&submit=Feeling+Lucky' % term]

Upvotes: 6

Related Questions