CODEWITHSUNDEEP
python-3.xscrapy
zwl1619
zwl1619

Reputation: 4232

Scrapy: After using `start_requests()` to make urls,`start_urls` is also necessary?

I am using Scrapy 1.1, my question is:

After using start_requests() to make urls,start_urls is also necessary?
Can I delete it?

For example：

class demoSpider(RedisSpider):
    name = "demospider"

    #Can I delete `start_urls`?
    redis_key = 'demospider:start_urls'
    start_urls = ['http://www.example.com']

    def start_requests(self):
        pages=[]
        for i in range(1,10):
            url='http://www.example.com/?page=%s'%i
            page=scrapy.Request(url)
            pages.append(page)
        return pages

Upvotes: 0

Views: 390

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21436

All scrapy spiders must inherit from scrapy.Spider (or it's children like scrapy.spiders.CrawlSpider) If you open this class up you'll see that self.start_urls is only used in start_requests method.

So if you override the inherited start_requests method, start_urls is no longer used anywhere and can be undefined or deleted.

Upvotes: 2

Related Questions